Serialization - How to do it properly!

Modules and scripting

Moderator: Garagecube

Serialization - How to do it properly!

Postby ghostofelvis100 » Fri Apr 27, 2012 2:14 pm

Hello Everyone.

When I started producing my own modules, Python was new to me. So, I did what I figured was the best method to get me going - copy source from other modules and begin. This method has served me well for many things, but beware, when copying someone else's source code, you are also copying their errors.

After much pain and late nights, I have discovered the following.


NAMING YOUR MODULE
This is critical. When a project is saved, it saves the module data according to the Name of each module. So if you update the name of your mod (eg. with a new version number in it), when you open a project file, Modul8 won't be able to find your module.

(I have just renamed all my modules in the online library, if you use any I suggest you update.)


HOW PYTHON STORES DATA
In Python, a statement like:
x = 1
print x
'1'
will create an integer holder and add the value of one into that holder.

if you then do this:
y = x
print y
'1'
what you are effectively saying is that y = the memory address of x

so,
x = 2
print y
'2'

but on the other hand if you do this:

x = 1
y = int(x)
x = 2
print y
'1'

with the above we have copied the value of x into a new integer holder y, so when we change the value of x, y remains unaffected.

HOW THIS APPLIES TO SAVING
The standard method of:
outDict['X'] = x
appears to work fine for single values, but when we get into more complex data holders like lists and dictionaries, it falls flat on its face.

Yes, if you have a enough memory on your machine you may be able to save and reload, and M8 will recover your data. But if you save, shutdown then restart, all your work will be lost.

Has anybody ever wondered what happened to all those lovely filters they set up in the Filter module, as you open up your project in front of a large audience? This is the reason.

DEEPCOPY
There is a python module called Copy, and it contains a function called deepcopy.
This function creates a deep copy of all the nested data. (a shallow copy being just the memory address)

to access this in your scripts:

Init Script:
import copy
from copy import deepcopy

exampleDict = {a:1, b:2, c:[1,2,3,4]}
------------------------
Serialize Script:
outDict['EXAMPLE_DICT'] = copy.deepcopy(exampleDict)
---------------------------
Deserialize Script:
exampleDict = copy.deepcopy(inDict['EXAMPLE_DICT'])

_____________________________________________________
I hope this is some help to everyone.

There is plenty of stuff written online about how to correctly assign data in python. Here I was hoping to give a brief explaination and show how it relates to Modul8.

I think all of my online modules where guilty of these errors in some way or another. Please upgrade as I have fixed them all.

Mods I have updated today and uploaded to the online library:
(at) BPM (global)
(at) Layer Sequencer 10x16
(at) Sound Router (layer)
(at) Sound Router (master)
(at) Filter (layer)
(at) LFO (global)
(at) LFO Router (layer)
(at) LFO Router (master)
User avatar
ghostofelvis100
activ8 member
 
Posts: 92
Joined: Mon Oct 25, 2010 10:19 pm
Location: Lampeter, UK

Re: Serialization - How to do it properly!

Postby anomad » Fri Apr 27, 2012 11:09 pm

. ghostofelvis100,

. thank you for the overview... i need to go through and fix some of my code as well. :)

. also (not to thread jack) - but i think it would be great for the fine folks @ garageCUBE to add a 'check for updated modules' feature automatically or user initiated that would look at the user module versions vs. what's online and offer to update them. right now, it's quite cumbersome to review all the existing modules and compare w/what's installed. it could even copy the older (local to the user) module into a subfolder, so they could back out if necessary.

. just a thought...

-james
(a nomad. )
anomad
master
 
Posts: 411
Joined: Sun Oct 21, 2007 10:07 pm
Location: north cakalacky, usa

Re: Serialization - How to do it properly!

Postby ghostofelvis100 » Sat Apr 28, 2012 12:03 am

Yes! That is a very good idea indeed.

At the moment all I can manage to do is post notes about updates here on the forum hoping people pick up on them.

However:
With this last bunch of updates I have made, I included a feature which checks a version number from an online text file, and then a message will appear on the modules to inform the user there is an update available.
The code is in the scripts and obviously anybody is welcome to copy it into their own mods.

This method does require module developers to host a file of version numbers, not a very user friendly system...

Also, there is no means for us to auto download and install updates, so yes, it does fall into Garagecube's hands to get this one sorted.

Nice one James!
User avatar
ghostofelvis100
activ8 member
 
Posts: 92
Joined: Mon Oct 25, 2010 10:19 pm
Location: Lampeter, UK

Re: Serialization - How to do it properly!

Postby sigmasix » Sat Apr 28, 2012 8:42 am

Wow, that is a very good post!!

Thanks a lot for this, this is very helpful!
User avatar
sigmasix
master
 
Posts: 1206
Joined: Thu Dec 02, 2004 2:12 pm
Location: gva | switzerland

Re: Serialization - How to do it properly!

Postby Lupin » Sat Apr 28, 2012 1:33 pm

Yeah thanks for this awesome post.
:D
User avatar
Lupin
ultim8 member
 
Posts: 381
Joined: Fri Aug 18, 2006 12:37 pm
Location: France

Re: Serialization - How to do it properly!

Postby Lupin » Sat Apr 28, 2012 1:42 pm

Sorry Andy, but when I refresh the modules Library , i can't find the recent update of the filter module the last one on the public library is the 1.6.1 (14/01/2011)
User avatar
Lupin
ultim8 member
 
Posts: 381
Joined: Fri Aug 18, 2006 12:37 pm
Location: France

Re: Serialization - How to do it properly!

Postby ghostofelvis100 » Sat Apr 28, 2012 2:28 pm

Yeah, having difficulty uploading that file (it's happened before with that mod!)

In the meantime, here is a zip file of my mods (Filter mod included), for manual installation.

http://blackearthmedia.co.uk/collected/m8/(at)%20M8%20Modules.zip
User avatar
ghostofelvis100
activ8 member
 
Posts: 92
Joined: Mon Oct 25, 2010 10:19 pm
Location: Lampeter, UK

Re: Serialization - How to do it properly!

Postby Lupin » Sat Apr 28, 2012 2:40 pm

thanks for all
User avatar
Lupin
ultim8 member
 
Posts: 381
Joined: Fri Aug 18, 2006 12:37 pm
Location: France

Re: Serialization - How to do it properly!

Postby Lupin » Fri Feb 24, 2017 1:28 pm

Hi Andy you can improve your code a bit
when you import a method from a class you don't need to recall the class


if you import the whole class copy then
your write

import copy

outDict['EXAMPLE_DICT'] = copy.deepcopy(exampleDict)


if you import deepcopy on the init script

from copy import deepcopy

then just call deepcopy
outDict['EXAMPLE_DICT'] = deepcopy(exampleDict)


wink:
User avatar
Lupin
ultim8 member
 
Posts: 381
Joined: Fri Aug 18, 2006 12:37 pm
Location: France

Re: Serialization - How to do it properly!

Postby ghostofelvis100 » Fri Feb 24, 2017 2:11 pm

I've often wondered that - guess I never tried it. Thanks.
User avatar
ghostofelvis100
activ8 member
 
Posts: 92
Joined: Mon Oct 25, 2010 10:19 pm
Location: Lampeter, UK


Return to Modules

Who is online

Users browsing this forum: No registered users and 2 guests

cron