EPB Format Documentation

Discussion in 'Empyrion API' started by cmwhee, Jun 13, 2017.

  1. cmwhee

    cmwhee Commander

    Joined:
    Oct 31, 2016
    Messages:
    118
    Likes Received:
    72
    Hey there,

    I've got an idea for a script that would create progressively "growing" versions of blueprint files. I'd like to make this script available to the community so they can see what bases look like at different stages, but it requires a lot of work just to digest the information that is in the EPB.

    a couple of people have created scripts that do similar things, but the quality of their code means that it really hard to expand on their work, which in many cases is buggy and incomplete.

    Is there any way that you guys can publish some of the system that serializes structures to EPB format? I don't even need the enum that describes which numbers correspond to which block type, all I really need are:

    * number of bytes in header (bonus if you can describe what the header actually means)
    * basic format of the block descriptions
    ** which bytes are block type
    ** which bytes are block position
    ** which bytes are block orientation
    ** (ideally) which bytes are texture/symbol maps
     
    #1
    Fractalite and jmcburn like this.
  2. Exacute

    Exacute Rear Admiral

    Joined:
    Feb 17, 2017
    Messages:
    456
    Likes Received:
    307
  3. geostar1024

    geostar1024 Rear Admiral

    Joined:
    Jan 24, 2016
    Messages:
    1,483
    Likes Received:
    2,459
    This is what I've got so far (haven't had time to work on it in a while):

    https://github.com/geostar1024/epbtools

    Basically, my code can parse the blueprint file and put all the block data (minus symbols, which are weird and I haven't figured them out yet) into an array. In principle, it can also write that block data back to the uncompressed format, but something isn't quite right yet (it doesn't match the original). The reason you can't edit blueprint files directly is because the actual block data is run through PKZIP before being appended to the file.

    EDIT: this requires python3; all the packages it requires should already be present in a python3 installation. My code isn't as neat as I'd like it, but there are comments at least!

    EDIT2: url updated
     
    #3
    Last edited: Aug 11, 2017
    Nahoi and Exacute like this.
  4. Exacute

    Exacute Rear Admiral

    Joined:
    Feb 17, 2017
    Messages:
    456
    Likes Received:
    307
    Byte 101 should be grpnameLength, 102+length, should be the name. Another is from 126, 127+length, which also seems to contain grpname

    That's all I had that you didn't.
     
    #4
  5. geostar1024

    geostar1024 Rear Admiral

    Joined:
    Jan 24, 2016
    Messages:
    1,483
    Likes Received:
    2,459
    I'm pretty sure I'm extracting the group names correctly; could you run it on one of your blueprints and see if it's working properly? The thing that I know I'm not handling right now are renamed devices; pretty sure my code will fail on such a blueprint . . .
     
    #5
  6. cmwhee

    cmwhee Commander

    Joined:
    Oct 31, 2016
    Messages:
    118
    Likes Received:
    72
    Man, you're a goddamn champion.

    Well commented code that handles everything in sequence. Thank you
     
    #6
  7. geostar1024

    geostar1024 Rear Admiral

    Joined:
    Jan 24, 2016
    Messages:
    1,483
    Likes Received:
    2,459
    You're welcome! If you learn anything about the rest of the stuff in the header, please post it here (or message me) so that I can update my code; alternately, you should be able to fork it and then issue a pull request against my repository.

    My overall goal is to first create 4 functions (read/write overall epb format, and read/write zipped block data) for working with epb files, and then implement some simple operations for working with blocks like copy, paste, and mass color/texture change, and then link all of this to a room generator to make procedurally-generated POIs (maybe ships too).

    EDIT: The embarrassing thing is how long it took me to figure out that the block data format was PKZIP; I spent hours trying to tease apart the raw hex, and then hours more looking into various compressed formats, before I happened to hit upon it.
     
    #7
    ZheBerg likes this.
  8. cmwhee

    cmwhee Commander

    Joined:
    Oct 31, 2016
    Messages:
    118
    Likes Received:
    72
    Yeah, umm as a nitpicky thing.

    PKZIP is software that was written byt the PKWARE software company

    https://en.wikipedia.org/wiki/PKZIP

    it used the ZIP file format to store data

    https://en.wikipedia.org/wiki/Zip_(file_format)

    which supports a bunch of encodings, in this case, I'm pretty sure (but not at all certain) the block data is compressed using the DEFLATE algorithm

    https://en.wikipedia.org/wiki/DEFLATE

    Awesome work though!

    edit: I have believed a lie for a very long time, PK and PKWARE are different companies
     
    #8
    Last edited: Jun 19, 2017
  9. geostar1024

    geostar1024 Rear Admiral

    Joined:
    Jan 24, 2016
    Messages:
    1,483
    Likes Received:
    2,459
    You are completely right on all accounts (and I should update that in my code). Did you get it to run, by the way?
     
    #9
  10. cmwhee

    cmwhee Commander

    Joined:
    Oct 31, 2016
    Messages:
    118
    Likes Received:
    72
    I haven't I'll be doing that when I get home tonight.
     
    #10
  11. Slipstream

    Slipstream Captain

    Joined:
    Sep 22, 2016
    Messages:
    429
    Likes Received:
    964
    Y'all are superheroes.
     
    #11
  12. geostar1024

    geostar1024 Rear Admiral

    Joined:
    Jan 24, 2016
    Messages:
    1,483
    Likes Received:
    2,459
    I'm pleased to announce that I have implemented preliminary support for both reading and writing blueprint files! The major limitation at present is that neither groups nor signal logic is extracted from the blueprint file, but everything else is. The resulting blueprint object can then be written back to a file that Empyrion can read. If you're going to modify blueprints with this, I'd definitely recommend reblueprinting the modified structure in-game after spawning it in before doing any more work on it.

    As the readme describes, I've implemented a couple of functions to do replacing of blocks. It's pretty rudimentary at present, but it does let you do a few things that aren't possible with the in-game console commands, like change all of one block shape into another.

    https://github.com/geostar1024/epbtools
     
    #12
  13. oojimaflip

    oojimaflip Captain

    Joined:
    Oct 6, 2016
    Messages:
    181
    Likes Received:
    535
    Awesome!
    Can't wait to see how this progresses ;)
     
    #13
  14. Nahoi

    Nahoi Ensign

    Joined:
    Aug 12, 2017
    Messages:
    4
    Likes Received:
    7
    I've done some work on this for the past few days and figured out a few more things like groups and signal logic.
    Here are my results so far:

    EDIT (2017-8-16) updated to include version 18

    Code:
    
    .epb file format
    
    All values are little endian unless specified otherwise.
    
    header:
        magic             4 bytes       hex: 45 52 94 78
        version           4 bytes       known versions: 15, 17, 18
        ship type         1 byte
        width             4 bytes
        heigh             4 bytes
        depth             4 bytes
        (unknown)       114 bytes
        timestamp         8 bytes
        (unknown)         9 bytes
        build             2 bytes
        (unknown)         3 bytes
        creator id      steamID[1]
        creator name    steamID
        owner id        steamID
        owner name      steamID
        (unknown)        11 bytes
        num lights        4 bytes
        (unknown)         4 bytes
        num devices       4 bytes
        (unknown)         4 bytes
        num blocks        4 bytes
        (unknown)         4 bytes       ! only in version 18 !
        num triangles     4 bytes
        num block types   2 bytes
        for each block type:
            block id      1 byte
            (unknown)     1 byte
            count         4 bytes
        (unknown)         1 byte
        num groups        2 bytes
        for each group:
            group name   string[2]
            (unknown)     1 byte
            (unknown)     1 byte       ! not present in version 15 !
            num devices   2 bytes
            for each device:
                position  4 bytes packed coordinates[3]
                name    string
        (unknown)         2 bytes
    
    The rest of the file is a zip archive, but the first two bytes are missing, they are: 0x50 0x4B ("PK")
    The zip archive contains a single file with the grid data.
    
    grid:
        The grid file starts with several bitfields. Each bitfield consists of:
            length      4 bytes           usually width*height*depth/8, rounded up
            bitfield     length bytes
            Then there is an entry for each "1" in the bitfield. The size of the entries varies between bitfields.
        blocks bitfield, entries:
            block id    1 byte
            rotation    1 byte
            subtype     2 bytes
        damage bitfield, entries:
            damage      2 bytes
        (unknown)       2 bytes
        color bitfield, entries:
            color       4 bytes
        texture bitfield, entries:
                        8 bytes
        symbol bitfield, entries:
                        4 bytes
        symbol rotation bitfield, entries:
                        4 bytes
     
        Next are several key-value-maps. The format of a map is:
            (unknown)       1 byte           seems to always be 1
            num entries       2 bytes
            for each entry:
                type       1 byte
                name       string
                value, depends on type:
                    0:   int, 4 bytes
                    1:   string
                    2:   1 byte
                    3:   float, 4 bytes
                    5:   color, 4 bytes
    
        The block data map contains additional data for some blocks, e.g. lcd panel text and color.
        num block data   2 bytes
        for each block data:
            position   4 bytes packed coordinates
            map, common entries are:
                "Text"   lcd panel text
                "ColB", "ColF"   lcd panel background/foreground
                "FS"   lcd panel font size
                "SizeFr", "SizeS1p", "SizeS1n", "SizeS2p", "SizeS2n"   sensor area size
        (unknown)       2 bytes
    
        The controls map contains a signal name for blocks that emit signals.
        num controls   2 bytes
        for each control:
            ! in version 15 !
                position   4 bytes packed coordinates
                name       string
            ! in version 17 and 18 !
                map, entries are:
                    "Pos"   block position, packed coordinates
                    "Name" 
                    "State"
    
        The signals map contains a list of targets for each signal.
        num signals       2 bytes
        for each signal:
            name       string
            num targets   2 bytes
            for each target:
                map, entries are:
                    "Pos"   block position, packed coordinates
                    "Func"
                    "Inv"
                    "Beh"
    
        The circuits map contains logic circuits like and/or.
        num circuits   2 byte
        for each circuit:
            map, entries can be:
                "OpName"
                "OutSig"
                "InSig0"
                "InSig1"
                "InSig2"
                "InSig3"
                "FPar"
    
    
    
    [1] steamID:
        (unknown)   5 bytes
        length      4 bytes big endian
        id          length bytes
    
    [2] string:
        The length of a string is encoded with variable length.
        The low 7 bits of each byte are part of the length, the high bit indicates wether another byte follows.
        Examples:
            20 (0x14) will be encoded as 0x14
            127 (0x7F) will be encoded as 0x7F
            128 (0x80) will be encoded as 0x80 0x01
            129 (0x81) will be encoded as 0x81 0x01
            32768 (0x8000) will be encoded as 0x80 0x80 0x02
        The length is followed by the string's content.
    
    [3] packed coordinates:
        Packed coordinates are 4 bytes (32 bits).
        Read as a little endian int, the layout is:
        | 1 | x, 11 bits | y, 8 bits | 1 | z, 11 bits |
        (there are two unused bits that seem to always be 1)
        To unpack the coordinates:
            x = (packed>>20)&0x7FF
            y = (packed>>12)&0xFF
            z = packed&0x7FF
    
    
    Oh and thanks geostar1024 for your work, it was a great starting point
     
    #14
    Last edited: Aug 16, 2017
  15. geostar1024

    geostar1024 Rear Admiral

    Joined:
    Jan 24, 2016
    Messages:
    1,483
    Likes Received:
    2,459
    Most excellent! As soon as I get the chance (possibly this weekend), I'll integrate your findings. Thanks especially for figuring out the signal logic and that the groups data contained packed coordinates (it would explain why I had so much trouble with the groups). Also/alternatively, if you feel like writing code, feel free to issue a pull request against the github repo.
     
    #15
    Exacute likes this.
  16. LiftPizzas

    LiftPizzas Rear Admiral

    Joined:
    Sep 28, 2015
    Messages:
    1,434
    Likes Received:
    2,573
    Does this mean you wouldn't be able to write a blueprint that had two blocks overlapping?

    (Or am I misunderstanding the meaning of bitfield? Because it sounds like they are using a 3-d bitmap of which locations do or don't have a block.)
     
    #16
  17. Nahoi

    Nahoi Ensign

    Joined:
    Aug 12, 2017
    Messages:
    4
    Likes Received:
    7
    You're understanding it correctly. No overlapping blocks (the game probably can't handle that anyway)
    You could, however, add a block to multiple groups… someone needs to experiment with things like that o_O

    By the way, I just downloaded the newest experimental build and there is a new version of the format (18) and all the stock blueprints seem to be converted…
     
    #17
    LiftPizzas likes this.
  18. geostar1024

    geostar1024 Rear Admiral

    Joined:
    Jan 24, 2016
    Messages:
    1,483
    Likes Received:
    2,459
    As far as the bitfield is concerned, for multi-block devices it'd just have a bunch of zeros for locations within the multi-block device.

    Actually, if you modify the block data to produce overlapping devices, the game will still let you spawn in that blueprint (though it tends to give me an internal error when doing so); I've turned a rectangular prism of steel into a bunch of overlapping T2 large generators, for instance. However, it doesn't seem to persist when reloading a save (which makes sense), so there's no practical way to have overlapping blocks in-game.
     
    #18
    oojimaflip, Nahoi and LiftPizzas like this.
  19. LiftPizzas

    LiftPizzas Rear Admiral

    Joined:
    Sep 28, 2015
    Messages:
    1,434
    Likes Received:
    2,573
    Ah yeah I guess if you can set the location of bitfields you could have those overlap.

    I wasn't looking to overlap devices as much as blocks. For example placing two (or four) of the thin walls in different orientations to make a hallway, overlapping blocks to make a transition, or placing a light in the same space as a block. Stuff like that. (Although I suppose windows and lights are devices, so I'd also want to overlap them LOL.)

    Oh well, it seems to be on many wishlists so maybe they'll make a legit way to do it. :)
     
    #19
  20. Nahoi

    Nahoi Ensign

    Joined:
    Aug 12, 2017
    Messages:
    4
    Likes Received:
    7
    Right, I forgot about multi-block devices.

    I imagine that would be very difficult to implement and require huge changes to the engine… then again, Medieval Engineers and some Minecraft mods have managed to do this already…
     
    #20

Share This Page