Zingg On Databricks #449

eric6204 · 2022-10-25T21:33:00Z

eric6204
Oct 25, 2022

I have tried running the FebrlExample.py and the ncVoters.py examples on Databricks in azure. In both cases I get an error when excuting: zingg.initAndExecute(). In both cases, I ensured the paths were correct and that all objects (args, options, pipes) were instantiated.

I am fairly new to Databricks and python so any help that you can give me to pinot me in the right direction would be appreciated.

Environment
zingg-0.3.4-py2.py3-none-any.whl
zingg-0.3.4-SNAPSHOT.jar

Databricks Runtime Version:
10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)

Py4JJavaError                             Traceback (most recent call last)
<command-2713124095585756> in <module>
      3 #Zingg execution for the given phase
      4 zingg = ZinggWithSpark(args, options)
----> 5 zingg.initAndExecute()

/databricks/python/lib/python3.8/site-packages/zingg/client.py in initAndExecute(self)
     52         """ Method to run both init and execute methods consecutively """
     53         self.client.init()
---> 54         self.client.execute()
     55 
     56     def getMarkedRecords(self):

/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1302 
   1303         answer = self.gateway_client.send_command(command)
-> 1304         return_value = get_return_value(
   1305             answer, self.gateway_client, self.target_id, self.name)
   1306 

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
    115     def deco(*a, **kw):
    116         try:
--> 117             return f(*a, **kw)
    118         except py4j.protocol.Py4JJavaError as e:
    119             converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling o603.execute.
: zingg.client.ZinggClientException
	at zingg.Trainer.execute(Trainer.java:69)
	at zingg.TrainMatcher.execute(TrainMatcher.java:51)
	at zingg.client.Client.execute(Client.java:242)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)

Answered by sonalgoyal

Oct 27, 2022

Thanks for this. I am able to run the findTrainingData phase of FebrlExample.py using ZinggWithSpark instead of Zingg in the example code. The reason it hasnt worked for you is that the example has phase trainMatch, which needs the model folder to be located in dbfs. If you can please 1. Change example to use ZinggWithSpark(you were right, this is the class to use with Databricks :D))
2. Upload models/100 to a location of your choice
3. Set zinggDir to adjust to the location of the models folder uploaded above
4. Run

View full answer

sonalgoyal · 2022-10-26T02:35:18Z

sonalgoyal
Oct 26, 2022
Maintainer

Thanks for trying Zingg @eric6204. Can you try changing ZinggWithSpark to Zingg?

1 reply

eric6204 Oct 26, 2022
Author

It originally was Zingg

zingg = Zingg(args, options)
zingg.initAndExecute()

an I got the same error so I thought that because I was in Databricks I needed to change it to ZinggWithSpark so that it could use the spark context. Here is the error stack from using Zingg(args, options) the remainder of the code is the same.

Py4JJavaError                             Traceback (most recent call last)
<command-2713124095585756> in <module>
      3 #Zingg execution for the given phase
      4 zingg = Zingg(args, options)
----> 5 zingg.initAndExecute()

/databricks/python/lib/python3.8/site-packages/zingg/client.py in initAndExecute(self)
     52         """ Method to run both init and execute methods consecutively """
     53         self.client.init()
---> 54         self.client.execute()
     55 
     56     def getMarkedRecords(self):

/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1302 
   1303         answer = self.gateway_client.send_command(command)
-> 1304         return_value = get_return_value(
   1305             answer, self.gateway_client, self.target_id, self.name)
   1306 

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
    115     def deco(*a, **kw):
    116         try:
--> 117             return f(*a, **kw)
    118         except py4j.protocol.Py4JJavaError as e:
    119             converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling o535.execute.
: zingg.client.ZinggClientException
	at zingg.Trainer.execute(Trainer.java:69)
	at zingg.TrainMatcher.execute(TrainMatcher.java:51)
	at zingg.client.Client.execute(Client.java:242)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)

sonalgoyal · 2022-10-26T18:26:47Z

sonalgoyal
Oct 26, 2022
Maintainer

Ok let me check this and get back, thanks for reporting

0 replies

sonalgoyal · 2022-10-26T18:29:12Z

sonalgoyal
Oct 26, 2022
Maintainer

Also, since you are calling the trainmatch job, do you already have the febrl model loaded on the dbfs? Can you please share your code and locations of csv and model folders in dbfs with their contents?

1 reply

eric6204 Oct 26, 2022
Author

Sure. I am running the example pulled directly from github. The only file referenced in the code are input files and they also exist in the github pull. It was pulled a s a zip file to my Databricks cluster.

wget https://github.com/zinggAI/zingg/releases/download/v0.3.3/zingg-0.3.3-SNAPSHOT-spark-3.1.2.tar.gz

Attached is the python file
FebrlExample.py.txt

I also notice that the fields definitions are for 10 fields but the input file has 11. could that be the issue?
test.csv

sonalgoyal · 2022-10-27T12:34:37Z

sonalgoyal
Oct 27, 2022
Maintainer

Thanks for this. I am able to run the findTrainingData phase of FebrlExample.py using ZinggWithSpark instead of Zingg in the example code. The reason it hasnt worked for you is that the example has phase trainMatch, which needs the model folder to be located in dbfs. If you can please 1. Change example to use ZinggWithSpark(you were right, this is the class to use with Databricks :D))
2. Upload models/100 to a location of your choice
3. Set zinggDir to adjust to the location of the models folder uploaded above
4. Run

5 replies

eric6204 Oct 27, 2022
Author

I made the changes and the process runs now, but it appears like it is trying to write back to the model directory. Is that correct? That the model is updated? Is there a way to suppress this? Or to point to a different location for the updated model to be written? My storage is immutable. So I am now getting this error:

Py4JJavaError                             Traceback (most recent call last)
<command-4108031075280848> in <module>
      1 #Zingg execution for the given phase
      2 zingg = ZinggWithSpark(args, options)
----> 3 zingg.initAndExecute()

/databricks/python/lib/python3.8/site-packages/zingg/client.py in initAndExecute(self)
     52         """ Method to run both init and execute methods consecutively """
     53         self.client.init()
---> 54         self.client.execute()
     55 
     56     def getMarkedRecords(self):

/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1302 
   1303         answer = self.gateway_client.send_command(command)
-> 1304         return_value = get_return_value(
   1305             answer, self.gateway_client, self.target_id, self.name)
   1306 

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
    115     def deco(*a, **kw):
    116         try:
--> 117             return f(*a, **kw)
    118         except py4j.protocol.Py4JJavaError as e:
    119             converted = convert_exception(e.java_exception)

/databricks/spark/python/lib/py4j-0.10.9.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    324             value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325             if answer[1] == REFERENCE_TYPE:
--> 326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
    328                     format(target_id, ".", name), value)

Py4JJavaError: An error occurred while calling o1639.execute.
: zingg.client.ZinggClientException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.
	at zingg.Trainer.execute(Trainer.java:69)
	at zingg.TrainMatcher.execute(TrainMatcher.java:51)
	at zingg.client.Client.execute(Client.java:242)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)

eric6204 Oct 27, 2022
Author

Ok after some more investigation that is the issue. the model is being updated and since it is stored in a location that is immutable I get this error. So either I have to turn off the update or move the model somewhere else.

Py4JJavaError: An error occurred while calling o2359.execute.
: zingg.client.ZinggClientException
	at zingg.Trainer.execute(Trainer.java:69)
	at zingg.TrainMatcher.execute(TrainMatcher.java:51)
	at zingg.client.Client.execute(Client.java:242)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)
shaded.databricks.org.apache.hadoop.fs.azure.AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.
	at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.delete(AzureNativeFileSystemStore.java:2828)
	at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.delete(AzureNativeFileSystemStore.java:2838)
	at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.deleteFile(NativeAzureFileSystem.java:2270)
	at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem$1.execute(NativeAzureFileSystem.java:2218)
	at shaded.databricks.org.apache.hadoop.fs.azure.AzureFileSystemThreadPoolExecutor.executeParallel(AzureFileSystemThreadPoolExecutor.java:223)
	at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.deleteWithInstrumentation(NativeAzureFileSystem.java:2228)
	at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1959)
	at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.delete(NativeAzureFileSystem.java:1932)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.$anonfun$delete$3(DatabricksFileSystemV2.scala:791)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.databricks.s3a.S3AExceptionUtils$.convertAWSExceptionToJavaIOException(DatabricksStreamUtils.scala:66)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.$anonfun$delete$2(DatabricksFileSystemV2.scala:789)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.$anonfun$withUserContextRecorded$2(DatabricksFileSystemV2.scala:1044)
	at com.databricks.logging.Log4jUsageLoggingShim$.$anonfun$withAttributionContext$1(Log4jUsageLoggingShim.scala:33)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:94)
	at com.databricks.logging.Log4jUsageLoggingShim$.withAttributionContext(Log4jUsageLoggingShim.scala:31)
	at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:205)
	at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:204)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionContext(DatabricksFileSystemV2.scala:510)
	at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:240)
	at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:225)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionTags(DatabricksFileSystemV2.scala:510)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withUserContextRecorded(DatabricksFileSystemV2.scala:1017)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.$anonfun$delete$1(DatabricksFileSystemV2.scala:787)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:330)
	at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:424)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:444)
	at com.databricks.logging.Log4jUsageLoggingShim$.$anonfun$withAttributionContext$1(Log4jUsageLoggingShim.scala:33)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:94)
	at com.databricks.logging.Log4jUsageLoggingShim$.withAttributionContext(Log4jUsageLoggingShim.scala:31)
	at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:205)
	at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:204)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionContext(DatabricksFileSystemV2.scala:510)
	at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:240)
	at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:225)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionTags(DatabricksFileSystemV2.scala:510)
	at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:419)
	at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:339)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.recordOperationWithResultTags(DatabricksFileSystemV2.scala:510)
	at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:330)
	at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:302)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.recordOperation(DatabricksFileSystemV2.scala:510)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.delete(DatabricksFileSystemV2.scala:787)
	at com.databricks.backend.daemon.data.client.DatabricksFileSystem.delete(DatabricksFileSystem.scala:156)
	at org.apache.spark.ml.util.FileSystemOverwrite.handleOverwrite(ReadWrite.scala:679)
	at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:167)
	at zingg.model.Model.save(Model.java:179)
	at zingg.Trainer.execute(Trainer.java:62)
	at zingg.TrainMatcher.execute(TrainMatcher.java:51)
	at zingg.client.Client.execute(Client.java:242)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
	at py4j.Gateway.invoke(Gateway.java:295)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:251)
	at java.lang.Thread.run(Thread.java:748)
Caused by: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This operation is not permitted on a non-empty directory.
	at hadoop_azure_shaded.com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
	at hadoop_azure_shaded.com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
	at hadoop_azure_shaded.com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:196)
	at hadoop_azure_shaded.com.microsoft.azure.storage.blob.CloudBlob.delete(CloudBlob.java:1054)
	at shaded.databricks.org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.delete(StorageInterfaceImpl.java:338)
	at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.safeDelete(AzureNativeFileSystemStore.java:2765)
	at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.delete(AzureNativeFileSystemStore.java:2820)
	... 64 more

sonalgoyal Oct 27, 2022
Maintainer

Yes you are right. If you delete the models/100/model folder and run, it should be able to build a new model and write it. Or you can change trainMatch to match and get the results in the output folder.

eric6204 Oct 27, 2022
Author

Thanks this works now!

sonalgoyal Oct 27, 2022
Maintainer

Great!!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Zingg On Databricks #449

{{title}}

Replies: 4 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Zingg On Databricks #449

eric6204 Oct 25, 2022

Replies: 4 comments · 7 replies

sonalgoyal Oct 26, 2022 Maintainer

eric6204 Oct 26, 2022 Author

sonalgoyal Oct 26, 2022 Maintainer

sonalgoyal Oct 26, 2022 Maintainer

eric6204 Oct 26, 2022 Author

sonalgoyal Oct 27, 2022 Maintainer

eric6204 Oct 27, 2022 Author

eric6204 Oct 27, 2022 Author

sonalgoyal Oct 27, 2022 Maintainer

eric6204 Oct 27, 2022 Author

sonalgoyal Oct 27, 2022 Maintainer

eric6204
Oct 25, 2022

Replies: 4 comments 7 replies

sonalgoyal
Oct 26, 2022
Maintainer

eric6204 Oct 26, 2022
Author

sonalgoyal
Oct 26, 2022
Maintainer

sonalgoyal
Oct 26, 2022
Maintainer

eric6204 Oct 26, 2022
Author

sonalgoyal
Oct 27, 2022
Maintainer

eric6204 Oct 27, 2022
Author

eric6204 Oct 27, 2022
Author

sonalgoyal Oct 27, 2022
Maintainer

eric6204 Oct 27, 2022
Author

sonalgoyal Oct 27, 2022
Maintainer