Remember that the render cost ("class") is a function of the number of devices, number of lights, and number of triangles. I'd guess that devices are separated from regular blocks because devices almost certainly have additional code attached to them, so a device's impact on performance is not just what it takes to render it.
Argh! As of 8.5 an extra byte has snuck in between the device group name and the unknown flags. Can anyone figure out how to account for this? (The file version number has not increased, so I am guessing that something in the file indicates if these are there or not) Code: ... nBlockCounts: 0300 Type Count 1801 14000000 2e02 01000000 9c01 09000000 unknown05: 05 nDeviceGroups: 0300 name: 07 63 7067 436f 7265 //"cpgCore" 00 <-------------------------- 01 ff nDevices: 0100 00180080 00 name: 08 63 7067 4c69 6768 74 //"cpgLight" 00 <-------------------------- 01 ff nDevices: 1400 00083080 08 4c69676874203133 //"Light 13" 00084080 08 4c69676874203134 //"Light 14" ...
I just started up work again on my version, and ran smack into this issue. I ended up handling it by reading as many bytes as it took to encounter the 0xFF byte that always seems to be there and then doing the appropriate conversion. As far as I can tell, the extra byte is always 0x00, so I'm currently reading it as a 16-bit integer if there are two bytes. After doing some testing, it really does seem to be the case that this data (1 byte pre-20, 2 bytes post-20) indicates whether or not a group was auto-generated: 256 (little-endian; 1 if big-endian), and zero if it was manually created or modified by the player in any way. Also, I did some more work on the datetime field, which appears right before the steam_id section. It's prefixed by 0x05 at location 0x86; the next 7 bytes hold the timestamp in hundreds of nanoseconds since 1827-09-25 18:40:30 (don't ask me why that's the epoch, but it works for all of the blueprints I've tested so far); you have to lop off the last byte (which always seems to be 0x88) and replace it with 0x00 before converting to a 64-bit unsigned integer (from there divide by 1e7 and add it to the epoch). But seriously, why this epoch?
I am pretty sure that I managed to read those as standard .Net DateTime objects - skipping one byte somewhere. Not all of them made sense though - as if they had a strange date where only the time part were used.
That would be consistent with what I've been seeing, since normally Microsoft uses 1900 as their epoch. And it'd be very interesting if you came across a blueprint for which the epoch I've experimentally determined does not give the correct date.
I haven't done much research since August, but all the creation dates seem fine. On the files I had in early August they range from 2016-12-23 to 2018-07-02. The meta data field that I labelled "UnknownMetax12" however, shows the date as 0001-06-12 and a reasonable time. Some files list this as 0001-01-01 00:00:00 ( all zero in the actual data) though. It is quite possible that this isn't actually a datetime - it could be just a time followed by some other data that just happens to be zero for all files... EDIT: This is what the .Net documentation says:
Minor update: I realised that one of the bytes in the device group list, the one just before the entry count, is the index of the custom switch in the main control panel. 0xff means that none of the custom switches affect this group, zero means that Custom1 controls it, one means that Custom2 controls it and so on. I have not yet looked in to the two new file format versions that has been released since I last looked. I am guessing that version 21 added support for the colour palette addition and that version 22 added support for logistics setup. There may, of course, be more to these...
I have done some major cleanup of the reader code. Now it can read all epb files I have without crashing. Some data is still not interpreted correctly, but many files are 100% parsed. This is mostly due to adding version checks and acknowledging different way to store some data in different file format versions. Several unknown bytes have been removed from the format definition and these bytes now fit into the surrounding sections. The new section "BlockTags", previously known as "Unknown06" is interesting. This section assigns tags to blocks. This ranges from light parameters through enemy spawners to vendors. The documentation has been updated to illustrate how these tags are set up for the various tasks. I also renamed the Signals, Logic and LogicOps sections, both in the code and the format documentation. Makes it a little bit easier to understand how the logic rules are stored in the file. A signal source is a switch or detector that creates a true/false value. Signals are then mapped to objects that listen to it. The signal path can optionally go through a signal operator to create complex logic systems. My program can not yet create all of these sections, it still creates files of version 20 with most of the old assumptions still in place. In order to be able to have the program create block tags or logic systems, some more work has to be done. Specifically, there needs to be a run-time data structure to hold the various sections after reading them and the most challenging part is to add ways to specify these through the interface. Perhaps the command line interface has reached it's limit in terms of creating blueprints...
Hmm, the huge amount of data added to EPB files of version 23 is confusing. BAJ_PharosSubmergedTank.epb, BA_TOPSpiceDestillery.epb and BA_TestBase.epb seem to look somewhat similar but BA_FillerTest.epb has an extra two bytes in front of every matrix that appear to contain only booleans for each possible block position. No, I need a break...
I am back to not throwing exceptions on any EPB-file I have. I think that I am reading the filler blocks correctly but there are plenty of other new data in some files that I don't see how to read yet. I also added reading of a new section of data after the names of the custom buttons in the zipped block. I don't know what these are for though. PrefabsStock/BA_Prefab_Tier7a.epb: Unknown08 (1) 0: 1804E08003146080 "Landing Support" Prefabs/BA_EpsilonLongRangeRadarDish.epb: Unknown08 (3) 0: 0E5C8081074C8081 "epsilon array" 1: 0E5CC082074C2081 "array 2" 2: 0ECCC081070CE080 "tower" The eight bytes look somewhat but not quite like two block pos references, but they appear to be out of range. Does anyone know what this is?
Added reading of the custom colour palettes. Now I get zero bytes left in the zipped data on all EPB files I have. That means that even if I don't know exactly what every byte does, I think that I have all counters and data sizes mapped out. The data block after the zipped data, however is still a bit fishy. There is a length indicator specifying additional data matrices, including the filler block matrix. However, some files contain additional data after this that I can not find a counter/indicator for. I noticed that in some files, this data contains a repeat of the colour palette exactly as stored in the zipped data. This leads me to believe that this data is relevant in the context of "- Fixed: Blueprint file size gets bigger than it should be" from the 9.3 release notes. Hopefully no files created with 9.3 and up will have any of this additional data. Extra data seen in EPB files with build versions 2149, 2154, 2155, 2156, 2159, 2160, 2161, 2162, 2165, 2189. No error seen in EPB files with build versions 2197, 2198. This supports my theory that the bug was corrected between builds 2189 and 2197 and that my code now reads every byte of every EPB file correctly. (Several pieces of the read bytes are still not understood, though)
Added the new block types from A9.4 experimental using a script that extracts these from Content/Configuration/Config_Example.ecf. I could still use an automated way of mapping variant names to block types if someone tells me how. Currently I have to do that mapping manually in a not-so-simple dictionary definition in the source code. (BlockTypeDefinitions in EpbBlock.cs)
Man, this looks great, with corect presentation and x-y-z rotation, it's about to become an awesome BP viewer...
@ion_storm : Thanks! Of course, I don't have the actual 3D assets, and I probably never will due to copyright thingies, unless I can extract them from the Unity resource files at run-time. I only display the blocks as semi-transparent cubes, currently it colours the entire block as the colour of the first face of the block in the EPB. I can extend this to colour the faces individually. In theory, I could also display textures and symbols, but then I am dangerously close to that copyright thing again, so I probably won't do that. What I really want is a schematic overview onto which I can overlay the connections from the logic. I have started to work on the logic editor, but currently it only drops all the signal, target and circuit nodes at random positions in the window. The nodes can be dragged around to make it easier to see what connects to what but I want to figure out how to do automatic graph layout stuff and possibly also add a way to save an extra meta-data file containing the layout of the nodes. Then I want to show the selected nodes in the 3D view with lines showing the connected nodes. This last part is probably easier than the graph layout stuff.
Funny when everything is already read and I just have to add a few lines to make the data appear on screen.
Ok, I committed a bunch of more code to the GitHub repository: EPBLab - the Windows application that the screenshots above are from ecf - A command line tool for extracting stuff from ECF files. I use this to extract the block types from Config_Example.ecf since the gawk script I made can't look up categories via references. P.S. Is there a public list somewhere where all block types are listed and categorised? The Config_Example.ecf file do not contain all block types, for example the voxel ore blocks, like VoxelCobalt - Id=85, are not in there. Feel free to test this out, but be aware that a lot of this is far from finished and might even be broken in some respects.
I said that I wasn't going to add the Empyrion 3D assets to my viewer and I won't, but I started to add some basic shapes for building blocks by manually entering vertices and triangles. Unfortunately I ran in to some issues concerning coordinate systems and rotations and now it feels as if my brain wants to leave me. I had the command line application generate a series of steel blocks using the "Corner Long D" variant in a row along the X-axis. Each block had a different rotation so that all rotations were represented. Then I added, and iteratively refined, geometry and rotations so that my view lined up with the view inside the game. Great. Then I opened a generated pyramid blueprint and two of the sloping sides were correct but the sides along the z-axis were flipped. Inverting the z-axis in my view makes the pyramid look correct but loading SV_Prefab_Tier6 shows that some blocks are rotated correctly while others are not. It is possible that all of this headache is caused by differences in the coordinate system orientations between Empyrion/Unity and Windows/Media3D but right now it feels as if I am too stupid to figure this out on my own. Would anyone be interested in helping me sort this mess out?
The only two that could be flipped, are the Z respectively Y axis. In Unity Z is the depth axis and Y is up. In Windows (WPF) it might be that Z is up. But not sure there. Just try to switch Z and Y and see, if that helps. /jmc
