Microsoft.Azure.Management.HDInsight.Job 2.0.0-preview

• • Microsoft.Azure.Common (>= 2.1.0 && < 3.0.0)
  • WindowsAzure.Storage (>= 6.0.0 && < 7.0.0)

Some parameters of the job submission objects have changed:

 *  HiveJobSubmissionParameters
      1.   The parameter “Arguments” now has type IList<string>.
      2.   The parameter “Files” now has type IList<string>.
      3.   The parameter “Defines” now has type IDictionary<string, string>.
   
 *  MapReduceJobSubmissionParameters
      1.   The parameter “Arguments” now has type IList<string>.
      2.   The parameter “Files” now has type IList<string>.
      3.   The parameter “LibJars” now has type IList<string>.
      4.   The parameter “Defines” now has type IDictionary<string, string>.
      5.   The parameter “CmdEnv” now has type IDictionary<string, string>.
   
 *  MapReduceStreamingJobSubmissionParameters
      1.   The parameter “Arguments” now has type IList<string>.
      2.   The parameter “Defines” now has type IDictionary<string, string>.
      3.   There is a new parameter “Files” with type IDictionary<string, string>.
   
 *  PigJobSubmissionParameters
      1.   The parameter “Arguments” now has type IList<string>.
      2.   The parameter “Files” now has type IList<string>.
   
 *  SqoopJobSubmissionParameters
      1.   The parameter “Files” now has type IList<string>.
      2.   There is a new parameter “LibDir” with type string.
   
 *  You no longer need to provide a StatusDir argument for job submission requests – one is automatically generated for you.

 *  The unused parameter “UserName” has been removed for all job submission objects. The actual username is automatically picked up from BasicAuthenticationCloudCredentials.

 *  The unused parameter “errorLog” has been removed for all job submission objects. Task log collection is not supported in Hadoop 2.0 or greater. See JIRAs HIVE-6063 and HIVE-6671.

The methods “DownloadJobTaskLogsAsync”, “DownloadJobTaskLogs”, “GetJobTaskLogSummary”, and “GetJobTaskLogSummaryAsync” have been removed. Task log collection is not supported in Hadoop 2.0 or greater. See JIRAs HIVE-6063 and HIVE-6671.

The constructor for HDInsightJobManagementClient now supports providing an HTTP client retry policy for Job REST requests as an optional parameter.

 *  Example:
      public HDInsightJobManagementClient(string clusterDnsName, BasicAuthenticationCloudCredentials credentials, RetryPolicy retryPolicy = null)

 *  If a retry policy is not provided, then a default retry policy (HDInsightJobManagementClient.HDInsightRetryPolicy) will be used.

The methods “GetJobOutput”(+Async) and “GetJobErrorLogsAsync”(+Async) now accept an AzureStorageAccess object as input for storage details instead of three string arguments.

 *  Examples:
      Task<Stream> GetJobOutputAsync(string jobId, IStorageAccess storageAccess, CancellationToken cancellationToken);
      Stream GetJobOutput(this IJobOperations operations, string jobId, IStorageAccess storageAccess)
      Task<Stream> GetJobErrorLogsAsync(string jobId, IStorageAccess storageAccess, CancellationToken cancellationToken);
      Stream GetJobErrorLogs(this IJobOperations operations, string jobId, IStorageAccess storageAccess)

 *  Create the AzureStorageAccess object by providing Azure storage account details:
      new AzureStorageAccess(TestUtils.StorageAccountName, TestUtils.StorageAccountKey, TestUtils.DefaultContainer)