dotnet-interactive assembly extension #517

suhsteve · 2020-05-13T17:32:35Z

This extension will add a a middleware to the dotnet-interactive session when this nuget is added via #r "nuget: Microsoft.Spark.Extensions.DotNet.Interactive".

For every code submission, the middleware will add the compiled assembly (before the current code has been processed) to the spark context by calling SparkContext.AddFile(...).

In addition to this, we also add any nuget package dependencies that the current session depends on to SparkContext.AddFile(...)

PackageHelper identifies dependent nuget packages and generates the necessary files for Microsoft.DotNet.DependencyManager.DependencyProvider

Please see #515 for additional details.

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/AssemblyKernelExtension.cs

imback82 · 2020-06-03T02:29:23Z

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/AssemblyKernelExtension.cs

+                        string assemblyName =
+                            AssemblyLoader.NormalizeAssemblyName(preCompilation.AssemblyName);
+                        string assemblyPath = Path.Combine(tempDir.FullName, $"{assemblyName}.dll");
+                        if (!File.Exists(assemblyPath))


Should we handle the case where the assemblyPath already exists?

I think it's safe to remove now that we have the command is SubmitCode check.

If this is an unexpected case, should we just throw and fail early?

Previously, other dotnet-interactive internal calls were hitting the middleware, and running the dll emitting logic and this caused it to try to emit duplicate assemblies. But I believe with the command is SubmitCode we should be okay. But to be safe we can readd the check and not run the middleware.

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/PackageHelper.cs

imback82 · 2020-06-03T03:04:06Z

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/PackageHelper.cs

+                if (!IsPathValid(nugetFile.FullName))
+                {
+                    // Copy the nuget file to a location that works with 
+                    // SparkContext.AddFiles(..)


why is this? (e.g., why wouldn't it work)?

The original nupkg file can be in a location like, C:\Users\user with spaces\.nuget\packages\package.name\version\package.name.version.nupkg and that would not work with SparkContext.AddFiles(..) with a spark version lower than 3.

imback82 · 2020-06-03T03:07:01Z

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/PackageHelper.cs

+
+                foreach (FileInfo file in files)
+                {
+                    if (!file.Exists || s_filesCopied.Contains(file.Name))


Would it be easy to debug the !file.Exists case if we ever hit it?

We can remove this check and trust DirectoryInfo.EnumerateFiles(..).

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/PackageResolver.cs

imback82 · 2020-06-04T04:34:23Z

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/PackageResolver.cs

+        public IEnumerable<ResolvedNuGetPackage> GetPackagesToCopy()
+        {
+            PackageRestoreContext restoreContext =
+                ((ISupportNuget)KernelInvocationContext.Current.HandlingKernel)


should ISupportNuget be passed into the constructor of this class? In that case, is this class also testable now (not sure how hard it is to mock ResolvedPackageReference)? And I think this function can move to PackageHelper? (I don't get the abstraction layers of PackageResolver vs PackageHelper)

Originally moved out logic from PackageHelper into PackageResolver to do unit tests. Couldn't mock ISupportNuget, because PackageRestoreContext.ResolvedPackageReferences isn't a virtual property.

Moved logic back into PackageHelper and instead wrapping the ISupportNuget ... PackageRestoreContext.ResolvedPackageReferences call.

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/AssemblyKernelExtension.cs

imback82

LGTM except for few minor/nit comments.

...oft.Spark.Extensions.DotNet.Interactive/Microsoft.Spark.Extensions.DotNet.Interactive.csproj

...harp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive.UnitTest/PackageHelperTests.cs

imback82 · 2020-06-12T01:25:52Z

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/PackageHelper.cs

+
+namespace Microsoft.Spark.Extensions.DotNet.Interactive
+{
+    internal class PackageHelper


Is there a better name than Package'Helper'? :)

hmm.. I'm bad with naming. How about Package'Resolver' ?

yea sounds good to me.

imback82 · 2020-06-12T01:31:29Z

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/AssemblyKernelExtension.cs

+            string assemblyName =
+                AssemblyLoader.NormalizeAssemblyName(compilation.AssemblyName);
+            assemblyPath = Path.Combine(dstPath, $"{assemblyName}.dll");
+            if (!File.Exists(assemblyPath))


I thought this was a critical error scenario no?

I don't think it's expected. Should we log an error or throw an exception ?

let's throw an exception to fail fast

imback82 · 2020-06-12T01:35:23Z

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/PackageHelper.cs

+
+        private string GetPathRelativeToPackages(string file, DirectoryInfo directory)
+        {
+            string strippedRoot = file


Can you add one example for this function please?

Should I move the examples that are at the callsite of GetPathRelativeToPackages(..) and add it to the method description instead ?

oh, you can just add it here.

just relevant example related to this function should be fine.

suhsteve · 2020-06-12T05:54:54Z

@imback82 addressed your comments. LMK if they look fine.

imback82 · 2020-06-12T05:57:40Z

...rp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive.UnitTest/PackageResolverTests.cs

+                        probingPaths) 
+                });
+
+            var packageResolver =


nit: no need to break now?

imback82

LGTM, thanks @suhsteve!

suhsteve · 2020-06-12T07:12:01Z

@imback82 actually don't merge yet. The ISupportNuget interface has been changed with the latest version of dotnet-interactive.

jonsequitur · 2020-06-12T15:01:00Z

Also, please be aware of and weigh in on: dotnet/interactive#512.

imback82 · 2020-06-12T20:33:29Z

@suhsteve the tests keep failing with timeouts (I retried 3 times). Can you check?

suhsteve · 2020-06-12T20:46:46Z

@suhsteve the tests keep failing with timeouts (I retried 3 times). Can you check?

I've seen this happen quite a few times with the recent PRs. I think something fishy is going on with the test pipeline.

imback82 · 2020-06-12T20:57:52Z

I don't see the failures from recent builds here? https://dev.azure.com/dnceng/public/_build?definitionId=459&_a=summary

suhsteve · 2020-06-12T22:22:03Z

I don't see the failures from recent builds here? https://dev.azure.com/dnceng/public/_build?definitionId=459&_a=summary

I think it's an issue with DeltaTable.Foreach , issue #333 . If we get lucky then it will use the same thread that has the activeThreadSession .

imback82 · 2020-06-12T22:45:07Z

Well, I saw multiple timeouts (>60 mins). Let's see what happens with the latest one.

imback82 · 2020-06-12T22:46:08Z

You can check attempt 3 and 4.

imback82 · 2020-06-12T22:47:11Z

I think we need to merge #524 soon.

elvaliuliuliu · 2020-06-12T22:59:56Z

I am also seeing the issue of DeltaTable.Foreach while testing backward compatibility. For instance, bc test failed in the recent PR #536 and it happened when testing #524 as well.

imback82 · 2020-06-12T23:14:29Z

The latest failed with Delta error. But if you look at the runtime, it could not have finished within one hour. You can compare the runtimes with https://dev.azure.com/dnceng/public/_build/results?buildId=679547&view=logs&j=ca395085-040a-526b-2ce8-bdc85f692774. Any changes in this PR could cause the regression?

suhsteve · 2020-06-12T23:32:39Z

There shouldn't be. The last 4 commits successfully passed tests.
643db70
a73fabb
914a576
2259289

The checkins from yesterday shouldn't have increased the running time.

imback82 · 2020-06-12T23:47:16Z

Maybe build agent changed? I see that E2E for Spark 2.3.x is taking more than 2 minutes instead of <1 minute even in master: https://dev.azure.com/dnceng/public/_build/results?buildId=685678&view=logs&j=ca395085-040a-526b-2ce8-bdc85f692774&t=c58e8551-6b20-5658-7a75-0283b3f97acb

suhsteve added 4 commits May 13, 2020 10:20

initial commit

7361063

add generated files to disposable temp directory

cd59f99

fixes

8e42081

clean up

bd47b67

suhsteve changed the title ~~[WIP] dotnet-interactive assembly extension~~ dotnet-interactive assembly extension May 15, 2020

suhsteve added 4 commits May 15, 2020 01:42

remove enable flag

f7ddf2e

cleanup

3286701

serialize metadata

6da419c

update csproj

d7960f9

imback82 mentioned this pull request May 23, 2020

Using with .NET Interactive notebooks locally #529

Closed

jonsequitur reviewed May 23, 2020

View reviewed changes

src/csharp/Extensions/Microsoft.Spark.Extensions.DotNet.Interactive/AssemblyKernelExtension.cs Outdated Show resolved Hide resolved

suhsteve force-pushed the dotnetinteractive_extension branch from 9bd0f0e to d7960f9 Compare May 29, 2020 07:19

suhsteve added 10 commits May 29, 2020 00:19

Merge branch 'master' into dotnetinteractive_extension

fcc4c05

cleanup.

c3f5d5d

set environment variable

9d44c4c

only serialize if there is data.

66804ef

updates to work in vscode.

6abb34c

rename method.

7d1dfae

add comments.

776b6a5

update summary.

bb40a8f

use default SparkSession.

864b3d0

create temp dir method.

427782b

imback82 reviewed Jun 3, 2020

View reviewed changes

suhsteve added 4 commits June 3, 2020 02:12

PR comments.

c4f6321

add header

fb1b76b

remove newline

9e72edb

cleanup.

7e07b75

imback82 reviewed Jun 4, 2020

View reviewed changes

imback82 assigned suhsteve Jun 4, 2020

imback82 added the enhancement New feature or request label Jun 4, 2020

suhsteve added 2 commits June 4, 2020 03:00

cleanup

a73fabb

thread safe

643db70

imback82 reviewed Jun 12, 2020

View reviewed changes

suhsteve added 3 commits June 11, 2020 22:50

rename PackageHelper.cs

5ecf819

rename

f2e6d0b

PR comments

e6a1758

PR comments.

8d66c25

imback82 reviewed Jun 12, 2020

View reviewed changes

imback82 previously approved these changes Jun 12, 2020

View reviewed changes

remove line breaks.

e75e8eb

suhsteve dismissed imback82’s stale review via e75e8eb June 12, 2020 06:07

suhsteve added 2 commits June 12, 2020 19:12

commit to force auzre pipeline

1ac0e51

commit to force auzre pipeline

828ab62

imback82 approved these changes Jun 13, 2020

View reviewed changes

imback82 merged commit 4be3dce into dotnet:master Jun 13, 2020

suhsteve deleted the dotnetinteractive_extension branch June 13, 2020 04:15

dotnet-interactive assembly extension #517

dotnet-interactive assembly extension #517

Uh oh!

Conversation

suhsteve commented May 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suhsteve Jun 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suhsteve Jun 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

imback82 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

suhsteve commented Jun 12, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imback82 left a comment

Choose a reason for hiding this comment

Uh oh!

suhsteve commented May 13, 2020 •

edited

Loading

suhsteve Jun 4, 2020 •

edited

Loading

suhsteve Jun 3, 2020 •

edited

Loading