Serialization - How to do it properly!
  • User avatar
    ghostofelvis100
    activ8 member
    Posts: 93
    Joined: Mon Oct 25, 2010 10:19 pm
    Location: Lampeter, UK
    Contact:

    Serialization - How to do it properly!

    by ghostofelvis100 » Fri Apr 27, 2012 2:14 pm

    Hello Everyone.

    When I started producing my own modules, Python was new to me. So, I did what I figured was the best method to get me going - copy source from other modules and begin. This method has served me well for many things, but beware, when copying someone else's source code, you are also copying their errors.

    After much pain and late nights, I have discovered the following.


    NAMING YOUR MODULE
    This is critical. When a project is saved, it saves the module data according to the Name of each module. So if you update the name of your mod (eg. with a new version number in it), when you open a project file, Modul8 won't be able to find your module.

    (I have just renamed all my modules in the online library, if you use any I suggest you update.)


    HOW PYTHON STORES DATA
    In Python, a statement like:
    x = 1
    print x
    '1'
    will create an integer holder and add the value of one into that holder.

    if you then do this:
    y = x
    print y
    '1'
    what you are effectively saying is that y = the memory address of x

    so,
    x = 2
    print y
    '2'

    but on the other hand if you do this:

    x = 1
    y = int(x)
    x = 2
    print y
    '1'

    with the above we have copied the value of x into a new integer holder y, so when we change the value of x, y remains unaffected.

    HOW THIS APPLIES TO SAVING
    The standard method of:
    outDict['X'] = x
    appears to work fine for single values, but when we get into more complex data holders like lists and dictionaries, it falls flat on its face.

    Yes, if you have a enough memory on your machine you may be able to save and reload, and M8 will recover your data. But if you save, shutdown then restart, all your work will be lost.

    Has anybody ever wondered what happened to all those lovely filters they set up in the Filter module, as you open up your project in front of a large audience? This is the reason.

    DEEPCOPY
    There is a python module called Copy, and it contains a function called deepcopy.
    This function creates a deep copy of all the nested data. (a shallow copy being just the memory address)

    to access this in your scripts:

    Init Script:
    import copy
    from copy import deepcopy

    exampleDict = {a:1, b:2, c:[1,2,3,4]}
    ------------------------
    Serialize Script:
    outDict['EXAMPLE_DICT'] = copy.deepcopy(exampleDict)
    ---------------------------
    Deserialize Script:
    exampleDict = copy.deepcopy(inDict['EXAMPLE_DICT'])

    _____________________________________________________
    I hope this is some help to everyone.

    There is plenty of stuff written online about how to correctly assign data in python. Here I was hoping to give a brief explaination and show how it relates to Modul8.

    I think all of my online modules where guilty of these errors in some way or another. Please upgrade as I have fixed them all.

    Mods I have updated today and uploaded to the online library:
    (at) BPM (global)
    (at) Layer Sequencer 10x16
    (at) Sound Router (layer)
    (at) Sound Router (master)
    (at) Filter (layer)
    (at) LFO (global)
    (at) LFO Router (layer)
    (at) LFO Router (master)
  • anomad
    master
    Posts: 411
    Joined: Sun Oct 21, 2007 10:07 pm
    Location: north cakalacky, usa
    Contact:

    Re: Serialization - How to do it properly!

    by anomad » Fri Apr 27, 2012 11:09 pm

    . ghostofelvis100,

    . thank you for the overview... i need to go through and fix some of my code as well. :)

    . also (not to thread jack) - but i think it would be great for the fine folks @ garageCUBE to add a 'check for updated modules' feature automatically or user initiated that would look at the user module versions vs. what's online and offer to update them. right now, it's quite cumbersome to review all the existing modules and compare w/what's installed. it could even copy the older (local to the user) module into a subfolder, so they could back out if necessary.

    . just a thought...

    -james
    (a nomad. )
  • User avatar
    ghostofelvis100
    activ8 member
    Posts: 93
    Joined: Mon Oct 25, 2010 10:19 pm
    Location: Lampeter, UK
    Contact:

    Re: Serialization - How to do it properly!

    by ghostofelvis100 » Sat Apr 28, 2012 12:03 am

    Yes! That is a very good idea indeed.

    At the moment all I can manage to do is post notes about updates here on the forum hoping people pick up on them.

    However:
    With this last bunch of updates I have made, I included a feature which checks a version number from an online text file, and then a message will appear on the modules to inform the user there is an update available.
    The code is in the scripts and obviously anybody is welcome to copy it into their own mods.

    This method does require module developers to host a file of version numbers, not a very user friendly system...

    Also, there is no means for us to auto download and install updates, so yes, it does fall into Garagecube's hands to get this one sorted.

    Nice one James!
  • User avatar
    sigmasix
    master
    Posts: 1207
    Joined: Thu Dec 02, 2004 2:12 pm
    Location: gva | switzerland
    Contact:

    Re: Serialization - How to do it properly!

    by sigmasix » Sat Apr 28, 2012 8:42 am

    Wow, that is a very good post!!

    Thanks a lot for this, this is very helpful!
  • User avatar
    Lupin
    garageCube team
    Posts: 381
    Joined: Fri Aug 18, 2006 12:37 pm
    Location: France
    Contact:

    Re: Serialization - How to do it properly!

    by Lupin » Sat Apr 28, 2012 1:33 pm

    Yeah thanks for this awesome post.
    :D
  • User avatar
    Lupin
    garageCube team
    Posts: 381
    Joined: Fri Aug 18, 2006 12:37 pm
    Location: France
    Contact:

    Re: Serialization - How to do it properly!

    by Lupin » Sat Apr 28, 2012 1:42 pm

    Sorry Andy, but when I refresh the modules Library , i can't find the recent update of the filter module the last one on the public library is the 1.6.1 (14/01/2011)
  • User avatar
    ghostofelvis100
    activ8 member
    Posts: 93
    Joined: Mon Oct 25, 2010 10:19 pm
    Location: Lampeter, UK
    Contact:

    Re: Serialization - How to do it properly!

    by ghostofelvis100 » Sat Apr 28, 2012 2:28 pm

    Yeah, having difficulty uploading that file (it's happened before with that mod!)

    In the meantime, here is a zip file of my mods (Filter mod included), for manual installation.

    http://blackearthmedia.co.uk/collected/m8/(at)%20M8%20Modules.zip
  • User avatar
    Lupin
    garageCube team
    Posts: 381
    Joined: Fri Aug 18, 2006 12:37 pm
    Location: France
    Contact:

    Re: Serialization - How to do it properly!

    by Lupin » Sat Apr 28, 2012 2:40 pm

    thanks for all
  • User avatar
    Lupin
    garageCube team
    Posts: 381
    Joined: Fri Aug 18, 2006 12:37 pm
    Location: France
    Contact:

    Re: Serialization - How to do it properly!

    by Lupin » Fri Feb 24, 2017 1:28 pm

    Hi Andy you can improve your code a bit
    when you import a method from a class you don't need to recall the class


    if you import the whole class copy then
    your write

    import copy

    outDict['EXAMPLE_DICT'] = copy.deepcopy(exampleDict)


    if you import deepcopy on the init script

    from copy import deepcopy

    then just call deepcopy
    outDict['EXAMPLE_DICT'] = deepcopy(exampleDict)


    wink:
  • User avatar
    ghostofelvis100
    activ8 member
    Posts: 93
    Joined: Mon Oct 25, 2010 10:19 pm
    Location: Lampeter, UK
    Contact:

    Re: Serialization - How to do it properly!

    by ghostofelvis100 » Fri Feb 24, 2017 2:11 pm

    I've often wondered that - guess I never tried it. Thanks.

Who is online

Users browsing this forum: No registered users and 1 guest