One more Beginner Problem

I need to restore multiple tables in a sequence. I am writing separate data flow for each table restore.

If there is any error in any of the table restore data flow, I want to write that error in a log file.

I am writing a script component in every data flow, which will get error code & error description, that i am writing in the log fie.

Is there a way to create a public reusable error logging script file, which we can call from every data flow and log only errors which we want?

Also want to confirm if the way i am doing is correct way or is there any easier/better way to achieve this?

It goes something like this:

Try
'create sqlcommand specifying connection and query
'create reader

reader = sqlcommand.ExecuteReader() <== this throws an "Object reference not set to an instance of an object" exception

'... other stuff here...
Catch ex as Exception
Row.DirectToErrorOutput()
Finally
'dispose sqlcommand and reader here...
End Try

Despite the Catch block, the Script Component still fails, and doesn't get redirected to the ErrorOutput path.

Any ideas on how to solve this?sql

One Data Flow Task and Multiple Data Flows

I have a data flow task which has around 5 data flows (like the 2nd diagram shown here). These 5 simple flows with just a row count transformation in between. Now, I want to fail the entire task immediately even if one of the data flows failed. Right now if one flow fails the remaining flows fails after a long time, not immediately. How can I make it fails immediately.

The other I would like to do is Can I place these 5 data flows in a transaction, so that if one data flow fails, others data flows also roll backs? ( I assume its not possible)

Thanks

Hi Karunakaran,

I think the best way to accomplish what you're describing, assuming your destination is a database, would be to use a transansaction. You can cause your entire data flow to be placed in a single distributed transaction by setting the "TransactionOption" property on your Data Flow Task to "Required". This will cause the data flow to attempt to enlist each of the connections it uses into a single transaction.

-David

OLEDB Source Table Locks?

Hi All,

Is it possible that an OLEDB Data Flow Source is imposing locks on the source tables? The source is an SQL Server OLTP environment, and although the package will be scheduled to run nightly when the application sees little to no use, I want to be sure that the process isn't impacting any application functions.

Thanks for the advice!

Rocco

An OLE-DB source will impose the same locks as if you were running the SELECT satement yourself from any other tool, so normaly you would expect some shared locks to allow you to consistently read the data requested.

Using Profiler will show you details of the SQL statement, and you can also choose to show locks details in profier, just filter for the SSIS connection/machine/user/application to focus on SSIS generated information as opposed to regular application traffic.

OLEDB Destination question

Situation

I have a package with an execute SQL task that truncates the destination table as the first step in the control flow and a data flow task that reads data from a flat file and loads a sql server table.

Once in a while the package bombs because it cannot get access to the flat file. The end result is that the table is empty because the truncate runs first. Obviously, I need to address the file contention, but I was wondering how to address this issue in general since anything that causes the data flow to blow up would leave the table empty.

I would rather have the table with day old data than empty, since it is not mission critical and the users can at least look at yesterday's data as opposed to nothing.

Question

Is there a way to specify a "load replace" on the OLEDB destination? I haven't seen one and I guess it makes sense because the data flow task transformations run row by row.

The only solution that I have come up with is to have the following on the control flow:

1) data flow task which reads flat file and loads a temp table

2) execute sql task to truncate the "real" destination

3) data flow task to move data from temp table to real table.

Anyone else come up with a better way to handle this?

Thanks!!

I guess from my perspective, I'd never truncate a table as the "first" step of an import process. You're guaranteed to have upset users when an issue that's completely out of your control (meaning, the missing flat file) asserts itself.

I'm using the following logic in a similar application:

1) In a Script Task, I verify the flat file is available, and contains at least one row of data.

2) If the "flat file verification" succeeds, a "Go_NoGo" boolean variable is set to True; otherwise, it remains False

3) An expression (@.[User::Go_NoGo]==True) on the constraint between the Script Task and the next task in the control flow effectively prevents the truncate / replace operation from running if the variable is False.

The code to verify the existance of the flat file is pretty simple:

Private Function GetFileExists(ByVal ImportFile_Path As String) As Boolean
Dim fi As System.IO.FileInfo

Try

fi = New System.IO.FileInfo(ImportFile_Path)

If Not fi.Exists Then
'No point in continuing - cannot find file:
'Discard FileInfo object:
fi = Nothing
Return False
End If

If fi.Length = 0 Then
fi = Nothing
Return False
End If

'Discard FileInfo object:
fi = Nothing

Return True

Catch ex As Exception

Throw ex

End Try

End Function 'GetFileExists

The FileInfo "Length" property tells you how big the flat file is, so it's a pretty good measure of whether or not the flat file contains data.

Obviously, the two If statement can be combined with an Or. In my application I actually check the count of lines in the flat file, and save it to make sure I import all the rows by comparing the line count with the row count in the destination table:

Private Function GetLineCount(ByVal ImportFile_Path As String) As Long
Dim LineCount As Long
Dim sr As System.IO.StreamReader
Dim str As String
Dim i As Long

Try
sr = New System.IO.StreamReader(ImportFile_Path)

Do
str = sr.ReadLine

If Not str Is Nothing Then
If str.Length > 1 Then
LineCount += 1
End If

str = String.Empty

End If

Loop Until str Is Nothing

'Discard StreamReader object:
sr.Close()
sr = Nothing

Return LineCount

Catch ex As Exception

Throw ex

End Try

End Function

I don't run the GetLineCount function until I know the file is there and is more than zero length.

|||

Interesting approach....I may add something similar to my process.

Thanks for taking the time to provide an answer!

ole db source editor "preview" throwing an error

i have a data flow configured to use a ole db source. the ole db source uses an ole db connection manager pointing to the adventureworks db which suceeded when i tested its connection. the data access mode of the ole db source is "sql command". below is the sql command text:

SELECT SpecialOfferID, Description
DiscountPct, Type, Category, StartDate,
EndDate, MinQty, MaxQty, ModifiedDate
FROM Sales.SpecialOffer
WHERE ModifiedDate >= ? AND ModifiedDate < ?

this query uses 2 paramaters, each of which is mapped to a datetime variable which falls with the range of the ModifiedDate column as follows:

Parameter0: User::ExtractStartDate

Parameter1: User::ExtractStopDate

ExtractStartDate is set to 7/1/2001 and ExtractStopDate is set to 3/31/2004. however, i get the following error when i press the preview button in the ole db source editor: "there was an error displaying the preview. additional information: no value given for one or more required parameters (microsoft sql native client)".

as far as i can tell, i have the ole db source configured correctly. thus, i can't figure out why this error is being generated. has anyone else experienced this issue? if so, were you able to resolve it? is this a bug?

thanks in advance.

Hi Duane, once you use parameters in your query, the query will not be able to parse or preview. It's a pretty normal hassle you'll get used to :).

Brian

|||

Brian Knight wrote:

Hi Duane, once you use parameters in your query, the query will not be able to parse or preview. It's a pretty normal hassle you'll get used to :).

Brian

thanks for the reply. i wonder if this is a bug or a "feature".|||

Not sure it's really either. It was the same way in DTS. Essentially the provider can't parse the literal query. Probably the ultimate answer is to perform the preview and parse like Reporting Services does where it pops open a box asking for the variable values you'd like to parse. As it stands, SSIS or DTS doesn't have enough info to perform a parse.

Brian

ole db source editor "preview" throwing an error

i have a data flow configured to use a ole db source. the ole db source uses an ole db connection manager pointing to the adventureworks db which suceeded when i tested its connection. the data access mode of the ole db source is "sql command". below is the sql command text:

SELECT SpecialOfferID, Description
DiscountPct, Type, Category, StartDate,
EndDate, MinQty, MaxQty, ModifiedDate
FROM Sales.SpecialOffer
WHERE ModifiedDate >= ? AND ModifiedDate < ?

this query uses 2 paramaters, each of which is mapped to a datetime variable which falls with the range of the ModifiedDate column as follows:

Parameter0: User::ExtractStartDate

Parameter1: User::ExtractStopDate

ExtractStartDate is set to 7/1/2001 and ExtractStopDate is set to 3/31/2004. however, i get the following error when i press the preview button in the ole db source editor: "there was an error displaying the preview. additional information: no value given for one or more required parameters (microsoft sql native client)".

as far as i can tell, i have the ole db source configured correctly. thus, i can't figure out why this error is being generated. has anyone else experienced this issue? if so, were you able to resolve it? is this a bug?

thanks in advance.

Hi Duane, once you use parameters in your query, the query will not be able to parse or preview. It's a pretty normal hassle you'll get used to :).

Brian

|||

Brian Knight wrote:

Hi Duane, once you use parameters in your query, the query will not be able to parse or preview. It's a pretty normal hassle you'll get used to :).

Brian

thanks for the reply. i wonder if this is a bug or a "feature".|||

Not sure it's really either. It was the same way in DTS. Essentially the provider can't parse the literal query. Probably the ultimate answer is to perform the preview and parse like Reporting Services does where it pops open a box asking for the variable values you'd like to parse. As it stands, SSIS or DTS doesn't have enough info to perform a parse.

Brian

OLE DB Source Editor

hi,

I am using SSIS to extract data from sql server and import into MDB file. In
the process, under data flow task, I have used OLE DB Source Editor as source. Here
i have choosen SQL Command as mode of data population. In the box below i
have typed the following statements.

"Exec Site_Address"

I have used many temperory tables in this procedure.
When i run this procedure in the query analyzer window i get the desired data which has to be imported to an MDB. After typing the above statements and when i
click the button preview i can see the data. But when i click the
Columns.... i dont see anything there. I am unable to see any columns there.
This is getting to my nerves because, when i use OLE DB as Destination i am
unable to map the columns and i get an error.

I dont know how to solve this problem. cannot we map columns in temp tables .... or wat is it ?

Please help me to find a solution.

I will also paste the procedure code that i have used.

Create procedure Site_Address

as

begin

create table #Data_For_Site_Address_Table

(

unitid varchar(20),

city varchar(50),

cust_num varchar(40),

zip varchar(20),

CountryID varchar(20),

CreatedBy varchar(20)

)

-- tblcrdsiteaddress

insert into #Data_For_Site_Address_Table

select distinct * from

(select

(select top 1 fsu.ser_num

from fs_unit fsu

where ca.cust_seq <> 0 and fsu.cust_num = ca.cust_num

order by ca.city desc) as UnitID,ca.city,ca.cust_num,ca.zip,

CASE

WHEN ca.country like 'Luxembourg' THEN 'LU'

WHEN ca.country like 'Deutschland' THEN 'DE'

WHEN ca.country like 'Austria' THEN 'AT'

WHEN ca.country like 'Czech Republic' THEN 'CZ'

WHEN ca.country like 'Denmark' THEN 'DK'

WHEN ca.country like 'CHINA' THEN 'CN'

WHEN ca.country like 'CROATIA' THEN 'HR'

WHEN ca.country like 'Egypt' THEN 'EG'

WHEN ca.country like 'Germany' THEN 'DE'

WHEN ca.country like 'Hungary' THEN 'HU'

WHEN ca.country like 'Jordan' THEN 'JO'

WHEN ca.country like 'Korea, Republic Of' THEN 'KR'

WHEN ca.country like 'Poland' THEN 'PL'

WHEN ca.country like 'Switzerland' THEN 'CH'

WHEN ca.country like 'United Kingdom' THEN 'GB'

ELSE '- N/A -' END AS CountryID, CA.CreatedBy

from custaddr ca

) al

where unitid is not null

Select TT.Unitid as Short_Site_Name, TT.City as Site_Name,'N.A' as Street_Po_Box,TT.Zip as Postal_Code_City, Null as State_Region,

TT.CountryID as CountryID,Null as Zone, Null as Note, TT.CreatedBy as UserID, GetDate() as Date, 'A' as [Action]

From #Data_For_Site_Address_Table TT

END

Thanks.

Rgds,
Meher Krishna.V

I just did a quick search in the forum and found a couple threads dealing with SP being used inside of an OLE DB Source:

http://forums.microsoft.com/MSDN/Search/Search.aspx?words=ole+db+source+procedure&localechoice=9&SiteID=1&searchscope=forumscope&ForumID=80

This, particularly, seems to talk about same issue:

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=161617&SiteID=1

I hope it helps

|||

Hopefully this should explain the problem:

Using stored procedures inside an OLE DB Source component
(http://blogs.conchango.com/jamiethomson/archive/2006/12/20/SSIS_3A00_-Using-stored-procedures-inside-an-OLE-DB-Source-component.aspx)

-Jamie

|||

I knew I have read about a solution somewhere; I just could not remember the exact place.

Good stuff Jamie!

|||

Rafael Salas wrote:

I knew I have read about a solution somewhere; I just could not remember the exact place.

Good stuff Jamie!

Well it wasn't at the above link cos I only wrote it about an hour ago

-Jamie

|||So, probably I dreamed about it. |||

thanks Jaime... that was great.

The problem is solved when i used a function......

But there should be a way how to use a sproc right ? In another posts that you had written, saying SET FMTONLY can solve the issue. i tried using it, but it was futile.

Any clue how to use it ?

|||

meher666 wrote:

thanks Jaime... that was great.

The problem is solved when i used a function......

But there should be a way how to use a sproc right ? In another posts that you had written, saying SET FMTONLY can solve the issue. i tried using it, but it was futile.

Any clue how to use it ?

No, in some circumstances I just don't think it can be done. Where did I suggest SET FMTONLY?

-Jamie

|||

okieee.... i am sorry again.... i thought it was you suggested using that statement. It was somebody else.

Thanks for your help.

OLE DB Source - Data Access Mode - SQL Command

I got a package with data flow task. Within the data flow task I have flat file with Fiscal Calendar defined. I got another data source within the data flow task, which is OLE DB Source. I want to use SQL Command as data access mode. SQL similar to one the below is in there.

-

DECLARE @.startdate DATETIME
DECLARE @.enddate DATETIME
DECLARE @.date DATETIME
DECLARE @.id INT

SET @.startdate = '1993-09-26' --Change these to
SET @.enddate = '2010-09-25' --whatever you want
SET @.id = 1
SET @.date = DATEADD(dd, @.id, @.startdate)

WHILE @.date <= @.enddate
BEGIN
select @.date CalendarDate,
DATEPART(dd, @.date) CalendarDayMonth,
DATEPART(dy, @.date) CalendarDayYear,
DATEPART(dw, @.date) CalendarDayWeek,
DATENAME(dw, @.date) CalendarDayName

SET @.id = @.id + 1
SET @.date = DATEADD(dd, @.id, @.startdate)

END

-

This SQL works fine in SSMS and returns around 6000 rows. But when I plug the same SQL in OLE DB Source it returns only the first record. It is not going through the WHILE loop.

Has anyone came across this?

Thanks

Sutha

Hey,

OLEDB Source connection to what database?

Brian

|||

I am connecting to my warehouse DB but the source is just the SQL, it doesn't need to extract anything from DB, as the SQL should give the result set.

What I should ideally use is "Execute SQL Task", which is in Control Flow Task.

Maybe I could achieve this by putting into a temp table and source it from the temp table. I am going to check it out.

Thanks

Sutha

ole db destination not inserting data

my package has a data flow task that is attempting to insert data into a sql server table using the ole db destination. the package validates and runs without reporting any errors, but the data was not inserted into the table. my data mappings look fine. how can i see what's happening when the component attempts to insert the data into the table?

Can you post all log messages that are produced from your data-flow?

Regards

Jamie

|||

Jamie Thomson wrote:

Can you post all log messages that are produced from your data-flow?

Regards

Jamie

here's the info from the output windows:

SSIS package "CaseStudy_Load.dtsx" starting.
Information: 0x4004300A at Data Flow Lockbox Detail Data Task, DTS.Pipeline: Validation phase is beginning.
Warning: 0x80047076 at Data Flow Lockbox Detail Data Task, DTS.Pipeline: The output column "PaymentAmount" (294) on output "Merge Join Output" (268) and component "Merge Join Checks and Invoices" (265) is not subsequently used in the Data Flow task. Removing this unused output column can increase Data Flow task performance.
Information: 0x4004300A at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Validation phase is beginning.
Information: 0x4004300A at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Validation phase is beginning.
Information: 0x40043006 at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Data Flow Lockbox Validate File and Header Info, Flat File Lockbox [1]: The processing of file "c:\casestudy\lockbox\samplelockbox.txt" has started.
Information: 0x400490F4 at Data Flow Lockbox Validate File and Header Info, Lookup BankBatchID [373]: component "Lookup BankBatchID" (373) has cached 0 rows.
Information: 0x4004300C at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Execute phase is beginning.
Information: 0x402090DE at Data Flow Lockbox Validate File and Header Info, Flat File Lockbox [1]: The total number of data rows processed for file "c:\casestudy\lockbox\samplelockbox.txt" is 11.
Information: 0x402090DF at Data Flow Lockbox Validate File and Header Info, OLE DB Destination Error Log [326]: The final commit for the data insertion has started.
Information: 0x402090E0 at Data Flow Lockbox Validate File and Header Info, OLE DB Destination Error Log [326]: The final commit for the data insertion has ended.
Information: 0x40043008 at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Post Execute phase is beginning.
Information: 0x402090DD at Data Flow Lockbox Validate File and Header Info, Flat File Lockbox [1]: The processing of file "c:\casestudy\lockbox\samplelockbox.txt" has ended.
Information: 0x40043009 at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: Cleanup phase is beginning.
Information: 0x4004300B at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: "component "OLE DB Destination Error Log" (326)" wrote 0 rows.
Information: 0x4004300B at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: "component "Script Component to Capture BatchItems" (643)" wrote 0 rows.
Information: 0x4004300B at Data Flow Lockbox Validate File and Header Info, DTS.Pipeline: "component "Script Component to store variables" (437)" wrote 0 rows.
SSIS package "CaseStudy_Load.dtsx" finished: Success.
The program '[3448] CaseStudy_Load.dtsx: DTS' has exited with code 0 (0x0).

below is the information written to log.txt:

#Fields: event,computer,operator,source,sourceid,executionid,starttime,endtime,datacode,databytes,message|||When debugging in BIDS - what is the number of rows passed to destination (it is displayed on the path leading from a previous transform to the destination while running package in BIDS)?

If the number is 0, no data has reached destination - something might be wrong with tranformations. Try adding data viewers to understand what data flows between transforms.|||ok. thanks. i discovered that the data isn't flowing from the flat file source. i'll make a new thread with a more appropriate subject line so that more people will notice my issue.

OLE DB Destination = temp table

Is it possible to write a dtsx so that the destination of an "OLE DB
Destination" Data Flow Destination is a temp table that is created earlier
in the dtsx ?On Mar 13, 7:05 pm, "John Grandy" <johnagrandy-at-gmail-dot-com>
wrote:
> Is it possible to write a dtsx so that the destination of an "OLE DB
> Destination" Data Flow Destination is a temp table that is created earlier
> in the dtsx ?
You may want to repost on the DTS/SSIS message groups. They are:
microsoft.public.sqlserver.dts and
microsoft.public.sqlserver.integrationsvcs. Sorry I could not be of
greater assistance.
Regards,
Enrique Martinez
Sr. SQL Server Developer

OLE DB Destination - table name variable

Hello,

In my Data Flow I have a OLE DB Destination that needs to get the table name to write the data to dynamicaly from a variable I created.

So I select "table name or view name variable" from the Data access mode and select my variable below. So far so good, but when I click "ok" I get the following error message :

Error at Data Flow Task [OLE DB Destination [45]]: A destination table name has not been provided.
Exception from HRESULT: 0xC0202042 (Microsoft.SqlServer.DTSPipelineWrap)

Am I doing something wrong or is there another way to set a variable table name for the OLE DB Destination ?

Using a variable in this way works fine for me on RTM.

I can get that error if I do not supply a valid table table in my variable before trying to use it in the OLE-DB Destination, which in itself makes sense too.|||Thanks ... I forgot to give the variable a valid default value, now it works.

OLE DB Command with property expression

Hi,

I am trying to use an OLE DB Command to run a different SQL command for each row in the data flow. I have a script component that builds the SQL command and puts it in a data flow variable, and a property expression in the data flow mapping [OLE DB Command].[Sql Command] to that variable.

The problem is that the OLE DB Command has a validation error, saying that "the command text was not set for the command object", and it doesn't run.

Did I miss anything? should I tell the OLE DB Command that the SQL command would come from expression? or is it a bug?

Thanks.

This validation error can be ignored if you set DataFlow task's DelayValidation to true. However, if you don't set parameters correctly, you may get parameters not bound error later on.

The way I do this is to set up a "dummy" OLEDBCommand first, with the good parameter binding(s), then during execution, the SqlCommand will be replaced by the my real expression value at runtime time - This will work under the assumption that the column metadata does not change overtime, which means, although the SqlCommand will change at runtime, the parameter bindings will remain the same.

Pls try it out and let me know if you have further questions.

Thanks

Wenyang

|||

Thanks.

Unfortunately, my sql commands differ in metadata. I can set it so that the parameters are in the same order for all commands, but some of the commands will not use all the parameters.

anyway, I tried what you suggested with a specific command, but it still doesn't work. the SQL command is deleted whenever I save the package (before running it), and then I get the same validation error.

Isn't there a straight forward solution? I mean, the package knows I set a property expression, otherwise it wouldn't delete the SQL command upon save. If so, why does it through the error? is it a bug?

|||

. It is by design the column metadata has to be the same for each SqlCommand expression value. This is the case not only for OLE DB Command, but also for other dataflow components when using expressions in similar conditions.

. I tried once again, as long as the expression is set correctly, the SqlCommand's value will be set to the expression evaluation result after saving my pkg(before execution). To me there is no bug here. Which version you are on? Did you set "DelayValidation" to true? Can you try again and make sure you set your expression at DataFlow task's "expression" property properly?

Thanks

Wenyang

|||

Thanks, you helped me find (part of) the problem.

I had the sql command set (using a property expression) to a variable that gets its value only during the data flow execution from a script component. the default value for the variable was empty, and when I saved the package it put the empty value into SqlCommand, which is not a valid value.

Setting DelayValidation to true didn't help here, since at the beginning of the data flow execution the variable is still empty, and I get the same validation error at runtime.

What did help is putting a dummy default value to the variable. now it is running without validation errors.

but...

it doesn't change the property of the OLE DB command :-(

the variable gets a different value for each processed row (I check it with a script), but the OLE DB command still runs the default value assigned to it at the beginning.

any ideas?

|||

Your scenario should work as well. Please provide the SqlServer version you are on and the detailed steps of how you set up the expression for OleDbCommand's SqlCommand property and I'll see how I can help.

Thanks

Wenyang

OLE DB Command with property expression

Hi,

I am trying to use an OLE DB Command to run a different SQL command for each row in the data flow. I have a script component that builds the SQL command and puts it in a data flow variable, and a property expression in the data flow mapping [OLE DB Command].[Sql Command] to that variable.

The problem is that the OLE DB Command has a validation error, saying that "the command text was not set for the command object", and it doesn't run.

Did I miss anything? should I tell the OLE DB Command that the SQL command would come from expression? or is it a bug?

Thanks.

This validation error can be ignored if you set DataFlow task's DelayValidation to true. However, if you don't set parameters correctly, you may get parameters not bound error later on.

The way I do this is to set up a "dummy" OLEDBCommand first, with the good parameter binding(s), then during execution, the SqlCommand will be replaced by the my real expression value at runtime time - This will work under the assumption that the column metadata does not change overtime, which means, although the SqlCommand will change at runtime, the parameter bindings will remain the same.

Pls try it out and let me know if you have further questions.

Thanks

Wenyang

|||

Thanks.

Unfortunately, my sql commands differ in metadata. I can set it so that the parameters are in the same order for all commands, but some of the commands will not use all the parameters.

anyway, I tried what you suggested with a specific command, but it still doesn't work. the SQL command is deleted whenever I save the package (before running it), and then I get the same validation error.

Isn't there a straight forward solution? I mean, the package knows I set a property expression, otherwise it wouldn't delete the SQL command upon save. If so, why does it through the error? is it a bug?

|||

. It is by design the column metadata has to be the same for each SqlCommand expression value. This is the case not only for OLE DB Command, but also for other dataflow components when using expressions in similar conditions.

. I tried once again, as long as the expression is set correctly, the SqlCommand's value will be set to the expression evaluation result after saving my pkg(before execution). To me there is no bug here. Which version you are on? Did you set "DelayValidation" to true? Can you try again and make sure you set your expression at DataFlow task's "expression" property properly?

Thanks

Wenyang

|||

Thanks, you helped me find (part of) the problem.

I had the sql command set (using a property expression) to a variable that gets its value only during the data flow execution from a script component. the default value for the variable was empty, and when I saved the package it put the empty value into SqlCommand, which is not a valid value.

Setting DelayValidation to true didn't help here, since at the beginning of the data flow execution the variable is still empty, and I get the same validation error at runtime.

What did help is putting a dummy default value to the variable. now it is running without validation errors.

but...

it doesn't change the property of the OLE DB command :-(

the variable gets a different value for each processed row (I check it with a script), but the OLE DB command still runs the default value assigned to it at the beginning.

any ideas?

|||

Your scenario should work as well. Please provide the SqlServer version you are on and the detailed steps of how you set up the expression for OleDbCommand's SqlCommand property and I'll see how I can help.

Thanks

Wenyang

OLE DB Command Stage: Capturing Rejects

Hello group, I have a question regarding the OLE DB Command Stage. Currently, I am reviewing a Data Flow that runs in production. This Data Flow Inserts to the various dimension tables in our warehouse. For a particular dimension table, the flow is like this:

Read Source records for Product combinations LookUp Product combinations against the current dimProduct table (cached in memory) Rows not found are then subjected to another LookUp on the dimProduct table (not cached). This is to find any rows inserted during the current run Rows not found are then Inserted to dimProduct using a Stored Procedure invoked by an OLE DB Command Successful Inserts then continue on, Rejected Inserts should be captured to a Flat File on our server for review.

Apparently, this last step has never been successful at capturing Rejects. Obviously, we would want to review these records to find the reason for failure. We get an empty file.

Currently, in the Stored Procedure we are using logic like this:

IF @.PRODUCTCOUNT <> 0

BEGIN

RAISERROR ('DUPLICATE PRODUCT!', 10, 1)

RETURN

END

Questions:

Is the RAISERROR command going to give us Output? Can we implement the OUTPUT command in our Proc invocation? I have not found any documentation that says the OLE DB Command Stage supports Error logging (Although columns are available to be added in the Input/Output columns tab?) Should we be using another Stage to accomplish this?

Any thoughts are welcome, thanks for your time!

rg

IF @.PRODUCTCOUNT <> 0

BEGIN

RAISERROR ('DUPLICATE PRODUCT!', 10, 1)

RETURN

END

In my experience, the error disposition on the OLE DB Command does not work. At least I haven't been able to get it to work. It will ignore RAISERROR statements, yet fail the component on a divide by zero, but never redirect a row. Even if the error redirection did work, you wouldn't be able to get the error description you're trying to raise.

Happily, output parameters DO work (which still surprises me since it isn't documented and isn't really intuitive). I recommend you use an output parameter for the error description, assign it to a column in the data flow, and put a Conditional Split right after the OLE DB Command evaluating the column to roll your own error redirection.

|||

Jay, thanks for the tip. The research was leading me in the direction that the error disposition was less than robust. Thanks for the confirmation, I will pursue using OUTPUT parameters for this data flow.

Thanks for the help! I have much more experience with a different ETL toolset, so even the little things right now are a challenge.

OLE DB Command not updating

I have a data flow task, and in that is an OLE DB Source that upon success, connects to an OLE DB Command. The OLE DB Source runs a sql command from a variable. (I've tested the sql; it parses and returns values.) The OLE DB Command is a very simple update sql statement with one column mapping. When I execute the task, it says it is successful, but when I check the table, nothing has been updated. I have ensured the connection is successful, and I am connecting to the correct db. Any suggestions would be helpful! Thanks!

Use SQL Profiler to capture the SQL statements that are being sent to the database. Odds are there is something related to the parameter mapping that is preventing the command from doing what you need it to.

Another thing to check is to ensure that the commands are being executed against the database that you think they are. If you have multiple connection managers, this can be easy to do.

|||

I found the problem. The variable used in the OLE DB source I thought I had changed to call a new stored procedure. However, its value was not actually changing and it was calling the wrong procedure, returning no values. My next question is why when I change the value in the properties window, does the variable not take the change?

|||Is the variables EvaluateAsExpression property set to true? This would cause that behavior.|||

Lindsay wrote:

I found the problem. The variable used in the OLE DB source I thought I had changed to call a new stored procedure. However, its value was not actually changing and it was calling the wrong procedure, returning no values. My next question is why when I change the value in the properties window, does the variable not take the change?

Did SQL Profiler end up being useful?

|||

Phil, yes the EvaluateAsExpression property was set to True; how embarassing LOL. Thank you!!

|||

Matthew, I've never used SQL Profiler before today, and am not familiar with it. I must need to modify the trace, because it has been running for a few hours now (actually I totally forgot about it)!!

|||

Lindsay wrote:

Matthew, I've never used SQL Profiler before today, and am not familiar with it. I must need to modify the trace, because it has been running for a few hours now (actually I totally forgot about it)!!

Oh no! It's probably best to just stop the trace at this point.

I'm sorry - I didn't mean to make things more complex. If you start the trace right before you run the package, and then stop it right after the package completes, you can then look through a (relatively) small set of queries sent to the server. In t his context, one of them should leap out because it the same query repeating over and over again. There are many ways to filter the data before and after it's recorded, but this is often the quickest and easiest way to see what the client application (in this case, SSIS) is REALLY sending to the server.

OLE DB Command and Destination writing to the same table

Hi,

I have a data flow task that performs an "upsert" by directing successful rows from a Lookup to an OLE DB Command that updates rows and unsuccessful rows (Lookup error output) to an OLE DB Destination for insertion.

The problem is that execution hangs when both tasks update/insert into the same table (execution is still hung after 20 minutes). Modifying the OLE DB Destination to insert into a different table succeeds (execution completese within 2 minutes). Replacing the OLE DB Destination with a Row Count transformation also works.

Could this be due to a table-locking issue? Any suggestions?

Thanks
ray

Just to confirm you haven't set the table lock check box on the oledb destination?

|||

You might look at the slowly changing dimension task... I'm not that familiar witrh it.

A quick and easy solution would send one of the outputs, probably the updates to a raw file destination. Then read that raw file in a different data flow task and perform the update there.

Doing a lot of updates in a sqlCommand - row by row - can be time consuming and I usually try to find another way to do it using some kind of set based operation. For example, if I can identify unique properties of from the source data, a date range or a specific field value; I would use that to construct a sqlCommand that would update all of the records at one time.

|||

Here is 2 links to pages that talk about different techniques for doing an "upsert". This is the

first time I have heard that term. I like it!

This is my web site:
http://www.mathgv.com/sql2005docs/SSISTransformScriptETL.htm

This is Jamie Thompson's cool

blog:
http://blogs.conchango.com/jamiethomson/archive/2006/09/12/SSIS_3A00_-Checking-if-a-row-exists-and-if-it-does_2C00_-has-it-changed.aspx

Thanks,

Greg Van

Mullem

|||

rlee1003 wrote:

Hi,

I have a data flow task that performs an "upsert" by directing successful rows from a Lookup to an OLE DB Command that updates rows and unsuccessful rows (Lookup error output) to an OLE DB Destination for insertion.

The problem is that execution hangs when both tasks update/insert into the same table (execution is still hung after 20 minutes). Modifying the OLE DB Destination to insert into a different table succeeds (execution completese within 2 minutes). Replacing the OLE DB Destination with a Row Count transformation also works.

Could this be due to a table-locking issue? Any suggestions?

Thanks
ray

Yes, this is absolutely a locking (actually a blocking) issue. Execute sp_lock or sp_who2 to confirm it.

You can get around this problem by executing the insert and update operations in different data-flows. Use a raw file to pass data from one data-flow to another. This technique is covered (albeit for use in a differrent context) here:

Splitting order detail and order header information from one file into multiple tables
(http://blogs.conchango.com/jamiethomson/archive/2006/05/22/3974.aspx)

-Jamie

|||Oho, I get the question now. My previous reply was useful but a little off topic. Are you use the "Fast Load" option in your OLE DB Destination? Turning this off might help.

You could also uses 2 "OLE DB Command" components (running insert and update commands). That's what I'm doing so I know it works quite well.

Thanks,
Greg

OLE DB - ADO .NET

Does anybody know why in some cases you can use both connection types, in others you can't. Like for example, you start a data flow task with a Datareader Source which uses an ADO .NET connection manager, next you want to do a lookup and seems like the only type of connections you can use in a lookup table is the OLE DB.

So OLE DB is placed hard coded in a lookup task, and I thought things changed since the coming of OOP ...

I think ole db is ok

just when i use script component,i use ado.net

|||Your observation is correct -- Lookup currently only supports OLE DB connections. We hope to add ADO.NET connection support in the future.|||Ok