72 Commits

Author SHA1 Message Date
Caleb J. Gardner 6b5c830234 Actually fix build 2026-03-18 06:17:07 -05:00
Caleb J. Gardner 116234cf9c Fixed build 2026-03-18 06:15:52 -05:00
Caleb J. Gardner 4601e8f323 Updated build 2026-03-18 06:13:19 -05:00
Caleb J. Gardner 50cae8b63d Use zig packed versions of zlib-ng, lz4, and zstd.
Changed use_c_libs to use_zig_decomp so c libraries are now default
2026-03-18 05:24:58 -05:00
Caleb J. Gardner 8b8c9a772f Added -Ddebug build option
Re-added errors from settings xattr values
2026-03-09 00:16:19 -05:00
Caleb Gardner f563648b20 Merge pull request #4 from CalebQ42/inode_finish
Re-do much of extraction
2026-03-08 22:36:37 -05:00
Caleb Gardner 8d4f3d72f8 Merge branch 'main' into inode_finish 2026-03-08 22:34:50 -05:00
Caleb J. Gardner 0b6129f1ae Switch to zlib-ng 2026-03-08 22:15:28 -05:00
Caleb J. Gardner 7308faad36 Moved kv free's in setMetadata 2026-03-05 12:54:25 -06:00
Caleb J. Gardner c9499251f8 Moved lookup tables into separate struct to fix some race conditions
Fixed lingering issues due to zero work size InodeFinish
Fixed xattrs not applying due to the keys sometimes not being
null-terminated.
Updated performance numbers
2026-03-05 12:20:30 -06:00
Caleb J. Gardner d470ca98e3 Added --force to unsquashfs
Fixing race condition bugs (yay)
2026-03-05 07:04:24 -06:00
Caleb J. Gardner a606f5e11a Move extract logic to util/extract.zig to make it easier to read. 2026-03-05 03:03:37 -06:00
Caleb J. Gardner a4e23a840d Updated performance values in README.
Added ability to ignore xattrs & permissions.
Ignore setting xattr errors due to an unknown issues.
2026-03-05 03:03:34 -06:00
Caleb J. Gardner 3a10572953 Updated performance values in README.
Added ability to ignore xattrs & permissions.
Ignore setting xattr errors due to an unknown issues.
2026-03-04 13:28:29 -06:00
Caleb J. Gardner edfe919c1b Expirementation with a new way to finish threads. Currently not working. 2026-03-04 06:39:44 -06:00
Caleb J. Gardner 4515610082 Added xattr support (currently untested)
Use stack fallback allocator on extraction for better performance.
2026-03-04 03:24:30 -06:00
Caleb J. Gardner beca6a2ae6 Small tweaks 2026-03-01 02:27:54 -06:00
Caleb J. Gardner 760e11df0b Fixed github workflow 2026-02-28 22:38:47 -06:00
Caleb J. Gardner 5f629df47c alloc-ified many functions.
Updated README
2026-02-28 22:33:57 -06:00
Caleb J. Gardner b0160e005b Some fixes 2026-02-17 05:54:32 -06:00
Caleb J. Gardner b7b99325da Slight improvement to how permissions are applied to folders
unsquashfs Verbose flag
2026-02-13 01:35:20 -06:00
Caleb J. Gardner b22e4d003d Update times 2026-02-12 08:57:36 -06:00
Caleb J. Gardner a41c37fef4 Updated unsquashfs version print out 2026-02-12 06:54:20 -06:00
Caleb J. Gardner 2d079d77f7 Exclusive file creation 2026-02-12 05:19:25 -06:00
Caleb J. Gardner 48f4235875 Fixed threads == 0 causing single threaded extraction.
Set exclusive file creation
2026-02-12 05:08:16 -06:00
Caleb J. Gardner 567dea8a0b Re-fix Action name 2026-02-12 04:28:32 -06:00
Caleb J. Gardner f78e5c7386 Rever to previous Action 2026-02-12 04:27:06 -06:00
Caleb J. Gardner 81e975c0d9 Try doing upload in two separate steps 2026-02-12 04:22:20 -06:00
Caleb J. Gardner fd274a8072 Different try to upload release files 2026-02-12 04:18:19 -06:00
Caleb J. Gardner 50ae79637e Removed broken ZSTD error value 2026-02-12 04:06:18 -06:00
Caleb J. Gardner ee41dc7278 Added import of zstd_errors.h 2026-02-12 03:58:08 -06:00
Caleb J. Gardner c34acebf51 Only link lzo if allow_lzo 2026-02-12 03:46:36 -06:00
Caleb J. Gardner bdbda29d39 Fixed wrong zig action line 2026-02-12 03:42:48 -06:00
Caleb J. Gardner ca4e867ddc Use setup zig Action 2026-02-12 03:39:05 -06:00
Caleb J. Gardner b066835066 Update apt 2026-02-12 02:51:01 -06:00
Caleb J. Gardner 2363bd7d10 sudo apt Actions 2026-02-12 02:43:55 -06:00
Caleb J. Gardner e619005b77 Fix action 2026-02-12 02:40:37 -06:00
Caleb J. Gardner 28b44891b3 Forgot Zig itself to the build 2026-02-12 02:32:34 -06:00
Caleb J. Gardner 4829c802a3 Clean version string 2026-02-12 02:32:06 -06:00
Caleb Gardner 570db9632a GH Action to create a release when release tagged 2026-02-12 02:30:21 -06:00
Caleb J. Gardner 0076294675 Small fix to testing to check if libc is linked 2026-02-12 02:19:38 -06:00
Caleb J. Gardner fd9e3d595b A bit of README cleanup
Added allow_lzo build option due to lzo build issues
2026-02-11 06:57:52 -06:00
Caleb J. Gardner b8189490eb Small change/fix when applying permissions 2026-02-09 13:50:56 -06:00
Caleb J. Gardner 6adc1d5c0c Fixes for threaded extraction.
Archive.extract now uses threaded extraction
2026-02-09 13:45:38 -06:00
Caleb J. Gardner 5ec12b5786 Finished adding multi-threaded extraction.
Added option in unsquashfs to specify the number of threads used.
Changed some functions to accept an allocator instead of just using
Archive's
Fixed run_tests.sh due to new c libraries
2026-02-08 15:14:35 -06:00
Caleb J. Gardner b892adacd7 Some work for threaded extraction 2026-02-08 10:27:35 -06:00
Caleb J. Gardner 2760ad6ccb Finished adding (untested) C decompression libraries. 2026-02-08 06:52:14 -06:00
Caleb J. Gardner 61311433b9 unsquashfs --help 2026-02-07 17:20:03 -06:00
Caleb J. Gardner 053d64a954 Some cleanup to build zon.
Minor start on threaded extraction
2026-02-07 17:20:03 -06:00
Caleb Gardner 0e0222cd02 Fix formatting for build flags in README 2026-02-07 11:34:34 -06:00
Caleb Gardner 9c0dfbadc2 Merge pull request #2 from CalebQ42/extract_expirement
Extract expirement
2026-02-07 11:31:51 -06:00
Caleb J. Gardner db2fb4b9f2 Fixed compilation when using zig test.
Remove option for static since -static should suffice in most situations
2026-02-07 11:24:58 -06:00
Caleb J. Gardner 067eaa87c2 You can now set when building to use c or zig libraries. 2026-02-07 10:58:32 -06:00
Caleb J. Gardner b64a3ec44a Archive.extract now directly uses Inode instead of File. 2026-02-07 06:57:55 -06:00
Caleb J. Gardner 704215e1a9 Remove updateTimes because I was running into integer overlow issues. 2026-02-07 06:42:34 -06:00
Caleb J. Gardner bcfd983f8d Fixed a handful of errors when extracting.
Fixed issues with unsquashfs
2026-02-07 06:28:27 -06:00
Caleb J. Gardner 75502da1d0 Remove DecompMgr in favor of a much simpler fn ptr.
Moved more functionality to Inode instead of File.
Started doing some optimization around allocation.
Slight rework of ExtractionOptions.
2026-02-07 05:09:17 -06:00
Caleb J. Gardner a316ba569f Renamed File.dev to File.devNum 2026-02-06 23:22:21 -06:00
Caleb J. Gardner a0f3f45885 Unsquashfs 2026-02-06 09:51:27 -06:00
Caleb J. Gardner f771ef7623 Work on extraction 2026-02-06 06:56:28 -06:00
Caleb J. Gardner 0d2576f5ee Fixed tables getting re-created 2026-01-31 06:39:41 -06:00
Caleb J. Gardner a76803aad1 Comments! 2026-01-31 05:14:00 -06:00
Caleb J. Gardner 1ff1e91d5e Added ExtractCompleteArchive test 2026-01-29 22:23:46 -06:00
Caleb J. Gardner 2bcbc16613 Extraction Finished-ish 2026-01-29 06:46:52 -06:00
Caleb J. Gardner 3c98cf2cdb EXTRACTION 2026-01-28 11:19:38 -06:00
Caleb J. Gardner 2c392cf250 Fixed a few bugs preventing basic functions
Worked on extraction, including creating DataReader
Added proper access to id, fragment, and export tables
2026-01-28 06:55:01 -06:00
Caleb J. Gardner 5d4e7b1435 Fixed things 2026-01-25 07:06:30 -06:00
Caleb J. Gardner 7aed59b5b1 Some fixes 2026-01-17 05:30:19 -06:00
Caleb J. Gardner f3fb8a128f Inodes! ExtractionOptions! Files! Directories! 2026-01-16 06:53:10 -06:00
Caleb J. Gardner 23bb19644b Finished (?) decompression, metadata reader, and Table. 2026-01-15 11:22:02 -06:00
Caleb J. Gardner ed14f13d9a Some work on tests & metadata 2026-01-15 06:56:43 -06:00
Caleb J. Gardner 428f938c3a Restart (once again) 2026-01-15 06:40:59 -06:00
30 changed files with 2821 additions and 1 deletions
+31
View File
@@ -0,0 +1,31 @@
name: Release Build
on:
push:
tags:
- "v*.*.*"
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v6
- uses: mlugg/setup-zig@v2
- name: Install deps
run: sudo apt update && sudo apt install -y liblzma-dev liblzo2-dev
- name: Build normal version
run: zig build --release=fast -Duse_zig_decomp=true -Dversion=${{ github.ref_name }}
- name: Move zig build out
run: mv zig-out/bin/unsquashfs ./unsquashfs-x86_64-zig-libs
- name: Rebuild with C libraries
run: zig build --release=fast -Dversion="${{ github.ref_name }}"
- name: Move C build out
run: mv zig-out/bin/unsquashfs ./unsquashfs-x86_64-c-libs
- name: Release
uses: softprops/action-gh-release@v2
with:
prerelease: true
files: |
unsquashfs-x86_64-zig-libs
unsquashfs-x86_64-c-libs
+2
View File
@@ -1,2 +1,4 @@
testing/
.zig-cache/
zig-out/
+25
View File
@@ -0,0 +1,25 @@
// Project-local debug tasks
//
// For more documentation on how to configure debug tasks,
// see: https://zed.dev/docs/debugger
[
{
"label": "Build & Run",
"adapter": "CodeLLDB",
"request": "launch",
"build": {
"command": "zig",
"args": ["build", "-Duse_c_libs=true", "-Ddebug=true"],
},
"program": "zig-out/bin/unsquashfs",
"args": [
"--force",
"-d",
"testing/TestExtractUnsquashfs",
"testing/LinuxPATest.sfs",
],
},
]
+1 -1
View File
@@ -1,6 +1,6 @@
MIT License
Copyright (c) 2025 Caleb Gardner
Copyright (c) 2026 Caleb Gardner
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+75
View File
@@ -0,0 +1,75 @@
# zig-squashfs
This is my experiments to learn Zig. Might amount to something. Might not.
A library and application to decompress or view squashfs archives.
## Current State
Overall works, but currently is missing some features ([see below](#capabilities)) and has significantly slow performance compared to `unsquashfs` ([see below](#performance)).
## Build options
> `-Duse_c_libs=true`
Instead of using Zig's standard library for decompression, use the system's C libraries. Has the benefit of being much faster and enabling LZO and LZ4 decompression.
> `-Dallow_lzo=true`
Enable compiling with LZO decompression support. The LZO library currently has some issues with Zig when imported so it's easier to just disable it by default. Only has an effect when using `-Duse_c_libs=true`.
> `-Ddebug=true`
Sets various build options that make debugging easier. Specifically, debug optimization is forced, valgrind support is enabled, error tracing is enabled, stipping is disabled, and copmilation uses LLVM (this is due to some linking issues when on Debug optimization and is required for debugging tools such as `lldb`. In the future this may be removed from the debug flag).
> `-Dversion=0.0.0`
Sets the version of `unsquashfs` shown when `--version` is passed.
## Capabilities
Most features are present except for the following:
* When using Zig decompression libraries then lzo and lz4 compression types are unavailable. I don't _currently_ plan on spending the time to find and validate a library since neither is popular.
* When using C decompression libraries, lzo is not supported by default due to [some issues](#build-considerations). If it's needed it's trivial to fix, but it's easiest to just leave it disabled.
## Performance
This is some basic observation's I've made about this library's performance when compared to `unsquashfs`. Unless otherwise stated, most observations were made when extracting my test archive (which is fairly small and uses zstd compression) and with `--release=fast`.
Currently, my only performance checks are checking execution time, nothing deeper.
* Under ideal circumstances, my library is ~70% slower (.12s vs .20s).
* Using Zig decompression libraries *significantly* increases decompression time by ~600%. Under ideal circumstances.
* Performance improvements/regressions will be common. I'm still learning Zig.
Example Times:
* *unsquashfs, multi-threaded*: .12s
* *unsquashfs, single-threaded*: .13s
* *C-libs, single-threaded*: .45s
* *C-libs, multi-threaded*: .20s
* *Zig-libs, single-threaded*: 5.78s
* *Zig-libs, multi-threaded*: 1.08s
## Build considerations
Compilation without `use_c_libs` works completely fine, but Zig has issues with some symbols from the lzo library that needs to be manually fixed. In particular you need to fix the definitions for `lzo_bytep` and `lzo_voidp` to be `*u8` and `?*anyopaque` respectively. Due to this, you have to manually enable LZO decompression using `-Dallow_lzo=true` when building.
```zig
pub const lzo_bytep = @compileError("unable to translate C expr: unexpected token ''");
// /usr/include/lzo/lzoconf.h:148:9
pub const lzo_charp = @compileError("unable to translate C expr: unexpected token ''");
// /usr/include/lzo/lzoconf.h:149:9
pub const lzo_voidp = @compileError("unable to translate C expr: unexpected token ''");
```
to
```zig
pub const lzo_bytep = *u8;
// /usr/include/lzo/lzoconf.h:148:9
pub const lzo_charp = @compileError("unable to translate C expr: unexpected token ''");
// /usr/include/lzo/lzoconf.h:149:9
pub const lzo_voidp = ?*anyopaque;
```
+101
View File
@@ -0,0 +1,101 @@
const std = @import("std");
pub fn build(b: *std.Build) !void {
const use_zig_decomp = b.option(bool, "use_zig_decomp", "Use zig standard library for decompression.") orelse false;
const allow_lzo = b.option(bool, "allow_lzo", "Compile with lzo support") orelse false;
const debug = b.option(bool, "debug", "Enable options to make debugging easier.") orelse false;
const version_string_option = b.option([]const u8, "version", "Version of the library/binary");
const zig_squashfs_options = b.addOptions();
zig_squashfs_options.addOption(bool, "use_zig_decomp", use_zig_decomp);
zig_squashfs_options.addOption(bool, "allow_lzo", allow_lzo);
const target = b.standardTargetOptions(.{});
const optimize = b.standardOptimizeOption(.{});
const mod = b.addModule("zig_squashfs", .{
.root_source_file = b.path("src/root.zig"),
.target = target,
.optimize = if (debug == true) .Debug else optimize,
.link_libc = !use_zig_decomp,
.valgrind = debug,
.error_tracing = debug,
.strip = if (debug == true) false else null,
});
mod.addOptions("config", zig_squashfs_options);
if (!use_zig_decomp) {
var zlib_ng = b.dependency("zlib_ng", .{});
mod.linkLibrary(zlib_ng.artifact("zng"));
mod.linkSystemLibrary("lzma", .{ .preferred_link_mode = .static });
if (allow_lzo == true)
mod.linkSystemLibrary("minilzo", .{ .preferred_link_mode = .static });
var lz4 = b.dependency("lz4", .{});
mod.linkLibrary(lz4.artifact("lz4"));
var zstd = b.dependency("zstd", .{});
mod.linkLibrary(zstd.artifact("zstd"));
}
var version = version_string_option orelse "0.0.0-testing";
if (version[0] == 'v') version = version[1..];
const unsquashfs_options = b.addOptions();
unsquashfs_options.addOption(
std.SemanticVersion,
"version",
try std.SemanticVersion.parse(version),
);
var exe_mod = b.createModule(.{
.root_source_file = b.path("src/bin/unsquashfs.zig"),
.target = target,
.optimize = if (debug == true) .Debug else optimize,
.link_libc = !use_zig_decomp,
.imports = &.{
.{ .name = "zig_squashfs", .module = mod },
},
.valgrind = debug,
.error_tracing = debug,
.strip = if (debug == true) false else null,
});
exe_mod.addOptions("config", unsquashfs_options);
const exe = b.addExecutable(.{
.name = "unsquashfs",
.root_module = exe_mod,
.use_llvm = debug,
});
const lib = b.addLibrary(.{
.name = "squashfs",
.root_module = mod,
.use_llvm = debug,
});
b.installArtifact(lib);
b.installArtifact(exe);
const mod_tests = b.addTest(.{
.root_module = mod,
});
const run_mod_tests = b.addRunArtifact(mod_tests);
const exe_tests = b.addTest(.{
.root_module = exe.root_module,
});
const run_exe_tests = b.addRunArtifact(exe_tests);
const test_step = b.step("test", "Run tests");
test_step.dependOn(&run_mod_tests.step);
test_step.dependOn(&run_exe_tests.step);
// zls build check steps
const lib_check = b.addLibrary(.{
.name = "squashfs",
.root_module = mod,
});
const exe_check = b.addExecutable(.{
.name = "unsquashfs",
.root_module = exe_mod,
});
const check = b.step("check", "Check if unsquashfs compiles");
check.dependOn(&lib_check.step);
check.dependOn(&exe_check.step);
}
+28
View File
@@ -0,0 +1,28 @@
.{
.name = .squashfs,
.version = "0.0.6",
.fingerprint = 0x37ba29474b87f145, // Changing this has security and trust implications.
.minimum_zig_version = "0.15.2",
.dependencies = .{
.zlib_ng = .{
.url = "git+https://github.com/CalebQ42/zig-zlib-ng#5f2f02dfb28acca2517dacbbd09e9b987f57b133",
.hash = "zlib_ng-2.3.3-pre1-2HYS4ClFAABW8KlHMyBHtlNKE3V7kCS8wqfxawG7xeaa",
},
.zstd = .{
.url = "git+https://github.com/allyourcodebase/zstd.git?ref=1.5.7-1#e1a501be57f42c541e8a5597e4b59a074dfd09a3",
.hash = "zstd-1.5.7-1-KEItkAMwAAD6OKY3m0OOmXG7aL-aLUfrDqbP5J5oYapU",
},
.lz4 = .{
.url = "git+https://github.com/allyourcodebase/lz4.git?ref=1.10.0-6#41f52ab227caf9d48cf88c89a4d2946caa12b102",
.hash = "lz4-1.10.0-6-ewyzw-4NAAAWDpY4xpiqr4LQhZQAC0x_rGnW2iPh6jk2",
},
},
.paths = .{
"build.zig",
"build.zig.zon",
"src",
"LICENSE",
"README.md",
},
}
Executable
+10
View File
@@ -0,0 +1,10 @@
#!/bin/sh
zig test \
-lc \
-lz \
-llzma \
-lminilzo \
-llz4 \
-lzstd \
src/test.zig
+97
View File
@@ -0,0 +1,97 @@
//! A squashfs archive read from a file.
//! Can be used to directly access File's contents or extract to the filesystem.
const std = @import("std");
const File = std.fs.File;
const builtin = @import("builtin");
const Decomp = @import("decomp.zig");
const ExtractionOptions = @import("options.zig");
const Inode = @import("inode.zig");
const InodeRef = Inode.Ref;
const BlockSize = @import("inode_data/file.zig").BlockSize;
const SfsFile = @import("file.zig");
const Superblock = @import("super.zig").Superblock;
const Tables = @import("tables.zig");
const MetadataReader = @import("util/metadata.zig");
const OffsetFile = @import("util/offset_file.zig");
const XattrTable = @import("xattr.zig");
const config = if (builtin.is_test) .{
.use_zig_decomp = !builtin.link_libc,
.allow_lzo = false,
} else @import("config");
const Archive = @This();
alloc: std.mem.Allocator,
fil: OffsetFile,
decomp: Decomp.DecompFn,
super: Superblock,
tables: ?Tables = null,
/// Default settings using std.Thread.getCpuCount() threads and the minimum of 4gb or half of system memory for memory usage.
pub fn init(alloc: std.mem.Allocator, fil: File, offset: u64) !Archive {
var super: Superblock = undefined;
const red = try fil.pread(@ptrCast(&super), offset);
std.debug.assert(red == @sizeOf(Superblock));
try super.validate();
const off_fil: OffsetFile = .init(fil, offset);
const decomp: Decomp.DecompFn = switch (super.compression) {
.gzip => Decomp.gzipDecompress,
.lzma => Decomp.lzmaDecompress,
.xz => Decomp.xzDecompress,
.zstd => Decomp.zstdDecompress,
.lz4 => if (!config.use_zig_decomp) Decomp.cLz4 else return error.Lz4Unsupported,
.lzo => if (!config.use_zig_decomp and config.allow_lzo) Decomp.lzoDecompress else return error.LzoUnsupported,
};
return .{
.alloc = alloc,
.fil = off_fil,
.decomp = decomp,
.super = super,
};
}
pub fn deinit(self: *Archive) void {
if (self.tables != null)
self.tables.?.deinit();
}
pub fn inode(self: *Archive, alloc: std.mem.Allocator, num: u32) !Inode {
if (self.tables == null)
self.tables = try .init(alloc, self);
const ref = try self.export_table.get(num - 1);
var rdr = try self.fil.readerAt(ref.block_start + self.super.inode_start, &[0]u8{});
var meta: MetadataReader = .init(alloc, &rdr.interface, &self.decomp);
try meta.interface.discardAll(ref.block_offset);
return try .read(alloc, &meta.interface, self.super.block_size);
}
pub fn root(self: *Archive, alloc: std.mem.Allocator) !SfsFile {
if (self.tables == null)
self.tables = try .init(alloc, self);
var rdr = try self.fil.readerAt(self.super.root_ref.block_start + self.super.inode_start, &[0]u8{});
var meta: MetadataReader = .init(alloc, &rdr.interface, self.decomp);
try meta.interface.discardAll(self.super.root_ref.block_offset);
const in: Inode = try .read(alloc, &meta.interface, self.super.block_size);
return .init(self, in, "");
}
pub fn open(self: *Archive, alloc: std.mem.Allocator, path: []const u8) !SfsFile {
if (self.tables == null)
self.tables = try .init(alloc, self);
var root_fil = try self.root(alloc);
defer if (!SfsFile.pathIsSelf(path)) root_fil.deinit();
return root_fil.open(path);
}
pub fn extract(self: Archive, alloc: std.mem.Allocator, path: []const u8, options: ExtractionOptions) !void {
var rdr = try self.fil.readerAt(self.super.root_ref.block_start + self.super.inode_start, &[0]u8{});
var meta: MetadataReader = .init(self.alloc, &rdr.interface, self.decomp);
try meta.interface.discardAll(self.super.root_ref.block_offset);
const in: Inode = try .read(self.alloc, &meta.interface, self.super.block_size);
try in.extractTo(alloc, self, path, options);
}
+133
View File
@@ -0,0 +1,133 @@
const std = @import("std");
const Writer = std.Io.Writer;
const builtin = @import("builtin");
const config = @import("config");
const squashfs = @import("zig_squashfs");
//TODO: Add more options
const help_mgs =
\\
\\Usage: unsquashfs [options] <archive>
\\
\\Options:
\\ -d <location> Extract to the given location instead of "squashfs-root"
\\
\\ -o <offset> Start reading the archive at the given offset.
\\ -dx Don't set xattr values
\\ -dp Don't set permissions (includes setting uid & gid owner)
\\
\\ -p <threads> Specify how many threads to use. If no present or zero, the system's logical cores count is used.
\\ -v Verbose
\\
\\ --force Force extraction. If the destination already exists, it will be deleted.
\\
\\ --help Display this messages
\\ --version Display the version
\\
;
const errors = error{InvalidArguments};
var archive: []const u8 = "";
var extLoc: []const u8 = "squashfs-root";
var offset: u64 = 0;
var threads: u32 = 0;
var verbose: bool = false;
var ignore_xattrs: bool = false;
var ignore_permissions: bool = false;
var force: bool = false;
pub fn main() !void {
const alloc = std.heap.smp_allocator;
var stdout = std.fs.File.stdout();
var out = stdout.writer(&[0]u8{});
defer out.interface.flush() catch {};
try handleArgs(alloc, &out.interface);
if (archive.len == 0) {
try out.interface.print("You must provide a squashfs archive\n", .{});
try out.interface.print(help_mgs, .{});
return;
}
var fil: std.fs.File = try std.fs.cwd().openFile(archive, .{}); //TODO: Handle error gracefully.
defer fil.close();
var arc: squashfs.Archive = try .init(alloc, fil, offset); //TODO: Update when memory size matters. //TODO: Handle error gracefully.
defer arc.deinit();
const options: squashfs.ExtractionOptions = .{
.threads = if (threads == 0) try std.Thread.getCpuCount() else threads,
.verbose = verbose,
.verbose_writer = if (verbose) &out.interface else null,
.ignore_xattr = ignore_xattrs,
.ignore_permissions = ignore_permissions,
};
if (force)
try std.fs.cwd().deleteTree(extLoc);
try arc.extract(alloc, extLoc, options); //TODO: Handle error gracefully.
}
fn handleArgs(alloc: std.mem.Allocator, out: *Writer) !void {
var args = try std.process.argsWithAllocator(alloc);
defer args.deinit();
_ = args.next(); // args[0] is the application launch command.
while (args.next()) |arg| {
if (std.mem.eql(u8, arg, "-o")) {
const nxt = args.next();
if (nxt == null or nxt.?.len == 0) {
try out.print("-o must be followed by a number\n", .{});
return errors.InvalidArguments;
}
offset = std.fmt.parseInt(u64, nxt.?, 10) catch {
try out.print("-o must be followed by a number\n", .{});
return errors.InvalidArguments;
};
continue;
} else if (std.mem.eql(u8, arg, "-d")) {
const nxt = args.next();
if (nxt == null or nxt.?.len == 0) {
try out.print("-d must be followed by a location\n", .{});
return errors.InvalidArguments;
}
extLoc = nxt.?;
continue;
} else if (std.mem.eql(u8, arg, "-p")) {
const nxt = args.next();
if (nxt == null or nxt.?.len == 0) {
try out.print("-p must be followed by a number\n", .{});
return errors.InvalidArguments;
}
threads = std.fmt.parseInt(u32, nxt.?, 10) catch {
try out.print("-p must be followed by a number\n", .{});
return errors.InvalidArguments;
};
continue;
} else if (std.mem.eql(u8, arg, "-v")) {
verbose = true;
continue;
} else if (std.mem.eql(u8, arg, "-dx")) {
ignore_xattrs = true;
continue;
} else if (std.mem.eql(u8, arg, "-dp")) {
ignore_permissions = true;
continue;
} else if (std.mem.eql(u8, arg, "--force")) {
force = true;
continue;
} else if (std.mem.eql(u8, arg, "--version")) {
try out.print("zig-unsquashfs v", .{});
try config.version.format(out);
try out.print("\nBuilt using Zig {s} in {} mode\n", .{ builtin.zig_version_string, builtin.mode });
std.process.exit(0);
return;
} else if (std.mem.eql(u8, arg, "--help")) {
try out.print(help_mgs, .{});
std.process.exit(0);
return;
}
if (archive.len > 0) {
try out.print("you can only provide one file at a time\n", .{});
try out.print(help_mgs, .{});
return errors.InvalidArguments;
}
archive = arg;
}
}
+261
View File
@@ -0,0 +1,261 @@
//! Implementations for decompression.
//! TODO: change to vtable interface to allow for shared decompressors for better performance/resource usage.
const std = @import("std");
const Reader = std.Io.Reader;
const builtin = @import("builtin");
const config = if (builtin.is_test) .{
.use_zig_decomp = !builtin.link_libc,
.allow_lzo = false, // Change once LZO compilation is fixed
} else @import("config");
const c = @cImport({
@cInclude("zlib-ng.h");
@cInclude("lzma.h");
@cInclude("lz4.h");
@cInclude("zstd.h");
@cInclude("zstd_errors.h");
if (config.allow_lzo)
@cInclude("lzo/minilzo.h");
});
pub const CompressionType = enum(u16) {
gzip = 1,
lzma,
lzo,
xz,
lz4,
zstd,
};
pub const DecompFn = *const fn (alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize; // TODO: replace anyerror to definitive error types.
pub const gzipDecompress = if (!config.use_zig_decomp) cGzip else zigGzip;
fn zigGzip(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
var rdr: Reader = .fixed(in);
const buf = try alloc.alloc(u8, out.len);
defer alloc.free(buf);
var decomp = std.compress.flate.Decompress.init(&rdr, .zlib, buf);
return decomp.reader.readSliceShort(out);
}
fn cGzip(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
_ = alloc;
var out_len: usize = out.len;
const res = c.zng_uncompress(out.ptr, &out_len, in.ptr, in.len);
return switch (res) {
c.Z_OK => out_len,
c.Z_MEM_ERROR => error.NotEnoughMemory,
c.Z_BUF_ERROR => error.OutBufferTooSmall,
c.Z_DATA_ERROR => error.BadData,
else => error.UnknownResult,
};
}
pub const lzmaDecompress = if (!config.use_zig_decomp) cLzma else zigLzma;
fn zigLzma(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
var rdr: Reader = .fixed(in);
var decomp = try std.compress.lzma.decompress(alloc, rdr.adaptToOldInterface());
return decomp.read(out);
}
fn cLzma(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
_ = alloc;
var stream: c.lzma_stream = .{
.next_in = in.ptr,
.avail_in = in.len,
.next_out = out.ptr,
.avail_out = out.len,
};
var res = c.lzma_alone_decoder(&stream, in.len * 2);
switch (res) {
c.LZMA_OK => {},
c.LZMA_MEM_ERROR => return error.LzmaMemoryError,
c.LZMA_PROG_ERROR => return error.LzmaProgramError,
else => return error.UnknownResult,
}
defer c.lzma_end(&stream);
while (res == c.LZMA_OK)
res = c.lzma_code(&stream, c.LZMA_RUN);
return switch (res) {
c.LZMA_STREAM_END => stream.total_out,
c.LZMA_MEM_ERROR => error.LzmaMemoryError,
c.LZMA_MEMLIMIT_ERROR => error.LzmaMemoryLimit,
c.LZMA_FORMAT_ERROR => error.LzmaBadFormat,
c.LZMA_DATA_ERROR => error.LzmaDataCorrupt,
c.LZMA_BUF_ERROR => error.LzmaCannotProgress,
c.LZMA_PROG_ERROR => error.LzmaProgramError,
else => error.UnknownResult,
};
}
// pub const lzoDecompress = if (!config.use_zig_decomp) cLzo else zigLzo;
// fn zigLzo(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
// _ = alloc;
// _ = in;
// _ = out;
// return error.LzoUnsupported;
// }
pub fn cLzo(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
_ = alloc;
var res = c.lzo_init();
if (res != 0) return error.LzoInitFailed;
var out_len: usize = out.len;
res = c.lzo1x_decompress(in.ptr, in.len, out.ptr, &out_len, null);
return switch (res) {
c.LZO_E_OK => out_len,
c.LZO_E_ERROR => error.LzoError,
c.LZO_E_OUT_OF_MEMORY => error.LzoOutOfMemory,
c.LZO_E_NOT_COMPRESSIBLE => error.LzoNotCompressible,
c.LZO_E_INPUT_OVERRUN => error.LzoInputOverrun,
c.LZO_E_OUTPUT_OVERRUN => error.LzoOutputOverrun,
c.LZO_E_LOOKBEHIND_OVERRUN => error.LzoLookbehindOverrun,
c.LZO_E_EOF_NOT_FOUND => error.LzoEofNotFound,
c.LZO_E_INPUT_NOT_CONSUMED => error.LzoInputNotConsumed,
c.LZO_E_NOT_YET_IMPLEMENTED => error.LzoNotYetImplemented,
c.LZO_E_INVALID_ARGUMENT => error.LzoInvalidArgument,
c.LZO_E_INVALID_ALIGNMENT => error.LzoInvalidAlignment,
c.LZO_E_OUTPUT_NOT_CONSUMED => error.LzoOutputNotConsumed,
c.LZO_E_INTERNAL_ERROR => error.LzoInternalError,
else => error.UnknownResult,
};
}
pub const xzDecompress = if (!config.use_zig_decomp) cXz else zigXz;
fn zigXz(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
var rdr: Reader = .fixed(in);
var decomp = try std.compress.xz.decompress(alloc, rdr.adaptToOldInterface());
return decomp.read(out);
}
fn cXz(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
_ = alloc;
var stream: c.lzma_stream = .{
.next_in = in.ptr,
.avail_in = in.len,
.next_out = out.ptr,
.avail_out = out.len,
// .allocator = _, TODO: create a custom allocator based on alloc,
};
var res = c.lzma_stream_decoder(&stream, in.len * 2, 0);
switch (res) {
c.LZMA_OK => {},
c.LZMA_MEM_ERROR => return error.LzmaMemoryError,
c.LZMA_PROG_ERROR => return error.LzmaProgramError,
else => return error.UnknownResult,
}
defer c.lzma_end(&stream);
while (res == c.LZMA_OK)
res = c.lzma_code(&stream, c.LZMA_RUN);
return switch (res) {
c.LZMA_STREAM_END => stream.total_out,
c.LZMA_MEM_ERROR => error.LzmaMemoryError,
c.LZMA_MEMLIMIT_ERROR => error.LzmaMemoryLimit,
c.LZMA_FORMAT_ERROR => error.LzmaBadFormat,
c.LZMA_DATA_ERROR => error.LzmaDataCorrupt,
c.LZMA_BUF_ERROR => error.LzmaCannotProgress,
c.LZMA_PROG_ERROR => error.LzmaProgramError,
else => error.UnknownResult,
};
}
// pub const lz4Decompress = if (!config.use_zig_decomp) cLz4 else zigLz4;
// fn zigLz4(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
// _ = alloc;
// _ = in;
// _ = out;
// return error.Lz4Unsupported;
// }
pub fn cLz4(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
_ = alloc;
const res = c.LZ4_decompress_safe(in.ptr, out.ptr, @intCast(in.len), @intCast(out.len));
if (res > 0) return @abs(res); // TODO: Find out what error values it can return.
return error.Lz4DecompressFailed;
}
pub const zstdDecompress = if (!config.use_zig_decomp) cZstd else zigZstd;
pub fn zigZstd(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
var rdr: Reader = .fixed(in);
const buf = try alloc.alloc(u8, 1024 * 1024);
defer alloc.free(buf);
var decomp = std.compress.zstd.Decompress.init(&rdr, buf, .{});
return decomp.reader.readSliceShort(out) catch |err| {
return decomp.err orelse err;
};
}
fn cZstd(alloc: std.mem.Allocator, in: []u8, out: []u8) anyerror!usize {
_ = alloc;
const res = c.ZSTD_decompress(out.ptr, out.len, in.ptr, in.len);
if (c.ZSTD_isError(res) == 0) return res;
return switch (c.ZSTD_getErrorCode(res)) {
c.ZSTD_error_prefix_unknown => cZstdError.PrefixUnknown,
c.ZSTD_error_version_unsupported => cZstdError.VersionUnsupported,
c.ZSTD_error_frameParameter_unsupported => cZstdError.FrameParameterUnsupported,
c.ZSTD_error_frameParameter_windowTooLarge => cZstdError.FrameParameterWindowTooLarge,
c.ZSTD_error_corruption_detected => cZstdError.CorruptionDetected,
c.ZSTD_error_checksum_wrong => cZstdError.ChecksumWrong,
c.ZSTD_error_literals_headerWrong => cZstdError.LiteralsHeaderWrong,
c.ZSTD_error_dictionary_corrupted => cZstdError.DictionaryCorrupted,
c.ZSTD_error_dictionary_wrong => cZstdError.DictionaryWrong,
c.ZSTD_error_dictionaryCreation_failed => cZstdError.DictionaryCreationFailed,
c.ZSTD_error_parameter_unsupported => cZstdError.ParameterUnsupported,
c.ZSTD_error_parameter_combination_unsupported => cZstdError.ParameterCombinationUnsupported,
c.ZSTD_error_parameter_outOfBound => cZstdError.ParameterOutOfBound,
c.ZSTD_error_tableLog_tooLarge => cZstdError.TableLogTooLarge,
c.ZSTD_error_maxSymbolValue_tooLarge => cZstdError.MaxSymbolValueTooLarge,
c.ZSTD_error_maxSymbolValue_tooSmall => cZstdError.MaxSymbolValueTooSmall,
c.ZSTD_error_stabilityCondition_notRespected => cZstdError.StabilityConditionNotRespected,
c.ZSTD_error_stage_wrong => cZstdError.StageWrong,
c.ZSTD_error_init_missing => cZstdError.InitMissing,
c.ZSTD_error_memory_allocation => cZstdError.MemoryAllocation,
c.ZSTD_error_workSpace_tooSmall => cZstdError.WorkSpaceTooSmall,
c.ZSTD_error_dstSize_tooSmall => cZstdError.DstSizeTooSmall,
c.ZSTD_error_srcSize_wrong => cZstdError.SrcSizeWrong,
c.ZSTD_error_dstBuffer_null => cZstdError.DstBufferNull,
c.ZSTD_error_noForwardProgress_destFull => cZstdError.NoForwardProgressDestFull,
c.ZSTD_error_noForwardProgress_inputEmpty => cZstdError.NoForwardProgressInputEmpty,
else => cZstdError.Generic,
};
}
pub const cZstdError = error{
Generic,
PrefixUnknown,
VersionUnsupported,
FrameParameterUnsupported,
FrameParameterWindowTooLarge,
CorruptionDetected,
ChecksumWrong,
LiteralsHeaderWrong,
DictionaryCorrupted,
DictionaryWrong,
DictionaryCreationFailed,
ParameterUnsupported,
ParameterCombinationUnsupported,
ParameterOutOfBound,
TableLogTooLarge,
MaxSymbolValueTooLarge,
MaxSymbolValueTooSmall,
CannotProduceUncompressedBlock,
StabilityConditionNotRespected,
StageWrong,
InitMissing,
MemoryAllocation,
WorkSpaceTooSmall,
DstSizeTooSmall,
SrcSizeWrong,
DstBufferNull,
NoForwardProgressDestFull,
NoForwardProgressInputEmpty,
FrameIndexTooLarge,
SeekableIo,
DstBufferWrong,
SrcBufferWrong,
SequenceProducerFailed,
ExternalSequencesInvalid,
MaxCode,
};
+60
View File
@@ -0,0 +1,60 @@
//! Directory entry from the directory table.
const std = @import("std");
const Reader = std.Io.Reader;
const InodeType = @import("inode.zig").InodeType;
const Entry = @This();
const Header = extern struct { // use extern due to bad alignment with packed.
count: u32,
block_start: u32,
num: u32,
};
const RawEntry = packed struct {
offset: u16,
inode_offset: i16,
inode_type: InodeType,
name_size: u16,
};
block_start: u32,
block_offset: u16,
num: u32,
inode_type: InodeType,
name: []const u8,
pub fn readDir(alloc: std.mem.Allocator, rdr: *Reader, size: u32) ![]Entry {
var cur_red: u32 = 3; // start at 3 due to "." & ".." being counted in the dir size.
var hdr: Header = undefined;
var raw: RawEntry = undefined;
var out: std.ArrayList(Entry) = try .initCapacity(alloc, 100); // Start out with a decent capacity instead of needing to allocate per header.
errdefer out.deinit(alloc);
while (cur_red < size) {
try rdr.readSliceEndian(Header, @ptrCast(&hdr), .little);
cur_red += @sizeOf(Header);
try out.ensureUnusedCapacity(alloc, hdr.count + 1);
for (0..hdr.count + 1) |_| {
try rdr.readSliceEndian(RawEntry, @ptrCast(&raw), .little);
const name = try alloc.alloc(u8, raw.name_size + 1);
errdefer alloc.free(name);
try rdr.readSliceEndian(u8, name, .little);
const val = out.addOneAssumeCapacity();
val.* = .{
.block_start = hdr.block_start,
.block_offset = raw.offset,
.num = @abs(hdr.num + raw.offset),
.inode_type = raw.inode_type,
.name = name,
};
cur_red += @sizeOf(RawEntry) + raw.name_size + 1;
}
}
return out.toOwnedSlice(alloc);
}
pub fn deinit(self: Entry, alloc: std.mem.Allocator) void {
alloc.free(self.name);
}
+202
View File
@@ -0,0 +1,202 @@
//! A file/directory within the squashfs archive.
const std = @import("std");
const File = std.fs.File;
const WaitGroup = std.Thread.WaitGroup;
const Mutex = std.Thread.Mutex;
const Archive = @import("archive.zig");
const DirEntry = @import("dir_entry.zig");
const ExtractionOptions = @import("options.zig");
const Inode = @import("inode.zig");
const BlockSize = @import("inode_data/file.zig").BlockSize;
const DataReader = @import("util/data.zig");
const MetadataReader = @import("util/metadata.zig");
const FileError = error{
NotDirectory,
NotRegularFile,
NotSymlink,
NotDevice,
NotFound,
ExtractionPathExists,
};
const SfsFile = @This();
archive: *Archive,
inode: Inode,
name: []const u8,
/// Initialize a new File.
/// name is copied to the File so can be safely freed afterwards.
pub fn init(archive: *Archive, inode: Inode, name: []const u8) !SfsFile {
const new_name = try archive.allocator().alloc(u8, name.len);
@memcpy(new_name, name);
return .{
.archive = archive,
.inode = inode,
.name = new_name,
};
}
pub fn fromEntry(archive: *Archive, entry: DirEntry) !SfsFile {
var rdr = try archive.fil.readerAt(entry.block_start + archive.super.inode_start, &[0]u8{});
var meta: MetadataReader = .init(archive.allocator(), &rdr.interface, archive.decomp);
try meta.interface.discardAll(entry.block_offset);
const inode: Inode = try .read(archive.allocator(), &meta.interface, archive.super.block_size);
errdefer inode.deinit(archive.allocator());
return .init(archive, inode, entry.name);
}
pub fn deinit(self: SfsFile) void {
var alloc = self.archive.allocator();
alloc.free(self.name);
self.inode.deinit(alloc);
}
fn getEntries(self: SfsFile) ![]DirEntry {
return self.inode.dirEntries(self.archive.allocator(), self.archive.*);
}
pub fn ownerUid(self: SfsFile) !u16 {
return self.archive.id(self.inode.hdr.uid_idx);
}
pub fn ownerGid(self: SfsFile) !u16 {
return self.archive.id(self.inode.hdr.gid_idx);
}
pub fn permissions(self: SfsFile) u16 {
return self.inode.hdr.permissions;
}
pub fn isRegular(self: SfsFile) bool {
return switch (self.inode.hdr.inode_type) {
.file, .ext_file => true,
else => false,
};
}
/// The returned DataReader will no longer work if the File's deinit function is called
/// or, more specifically, it's inode's deinit function is called.
pub fn dataReader(self: SfsFile) !DataReader {
return self.inode.dataReader(self.archive);
}
pub fn isDir(self: SfsFile) bool {
return switch (self.inode.hdr.inode_type) {
.dir, .ext_dir => true,
else => false,
};
}
pub fn iterate(self: SfsFile) !Iterator {
if (!self.isDir()) return FileError.NotDirectory;
return .{
.entries = try self.getEntries(),
.archive = self.archive,
};
}
/// Open a sub-file/folder within a directory at the given path.
/// If path is "", ".", "/", or "./", this File is returned.
pub fn open(self: SfsFile, alloc: std.mem.Allocator, path: []const u8) !SfsFile {
if (!self.isDir()) return FileError.NotDirectory;
if (pathIsSelf(path)) return self;
// Recursively stip ending & leading path separators.
if (path[0] == '/') return self.open(path[1..]);
if (path[path.len - 1] == '/') return self.open(path[0 .. path.len - 1]);
const idx = std.mem.indexOf(u8, path, "/") orelse path.len;
const first_element = path[0..idx];
if (std.mem.eql(u8, first_element, ".")) return self.open(path[idx + 1 ..]);
const entries = try self.getEntries();
defer {
for (entries) |e| {
e.deinit(alloc);
}
alloc.free(entries);
}
var cur_slice = entries;
var split = cur_slice.len / 2;
while (cur_slice.len > 0) {
split = cur_slice.len / 2;
const comp = std.mem.order(u8, first_element, cur_slice[split].name);
switch (comp) {
.eq => {
var fil: SfsFile = try .fromEntry(self.archive, cur_slice[split]);
if (idx == path.len) {
return fil;
}
defer fil.deinit();
return fil.open(alloc, path[idx + 1 ..]);
},
.lt => cur_slice = cur_slice[0..split],
.gt => cur_slice = cur_slice[split + 1 ..],
}
}
return FileError.NotFound;
}
pub fn isSymlink(self: SfsFile) bool {
return switch (self.inode.hdr.inode_type) {
.symlink, .ext_symlink => true,
else => false,
};
}
pub fn symlinkPath(self: SfsFile) ![]const u8 {
if (!self.isSymlink()) return FileError.NotSymlink;
return switch (self.inode.data) {
.symlink => |s| s.target,
.ext_symlink => |s| s.target,
else => unreachable,
};
}
/// Check if the File is a block or character device.
pub fn isDevice(self: SfsFile) bool {
return switch (self.inode.hdr.inode_type) {
.block_dev, .char_dev, .ext_block_dev, .ext_char_dev => true,
else => false,
};
}
/// If the File is a block or character device, get's it's device number.
pub fn devNum(self: SfsFile) !u32 {
if (!self.isDevice()) return FileError.NotDevice;
return switch (self.inode.data) {
.block_dev, .char_dev => |d| d.dev,
.ext_block_dev, .ext_char_dev => |d| d.dev,
else => unreachable,
};
}
/// Extract the given File to the path. If File is a regular file, the path must be a directory or not exist.
/// If the gievn path is a folder, the File's contents will be extracted within.
pub fn extract(self: *SfsFile, alloc: std.mem.Allocator, path: []const u8, options: ExtractionOptions) !void {
return self.inode.extractToThreaded(alloc, self.archive, path, options);
}
/// Utility function.
pub fn pathIsSelf(path: []const u8) bool {
if (path.len == 0) return true;
if (path.len == 1 and (path[0] == '/' or path[0] == '.')) return true;
if (path.len == 2 and (path[0] == '.' and path[1] == '/')) return true;
return false;
}
pub const Iterator = struct {
entries: []DirEntry,
archive: *Archive,
idx: u32 = 0,
pub fn next(self: *Iterator) !?SfsFile {
if (self.idx >= self.entries.len) return null;
defer self.idx += 1;
return try SfsFile.fromEntry(self.archive, self.entries[self.idx]);
}
pub fn deinit(self: Iterator) void {
var alloc = self.archive.allocator();
for (self.entries) |e| {
e.deinit(alloc);
}
alloc.free(self.entries);
}
};
+189
View File
@@ -0,0 +1,189 @@
//! A file-system object. Represents a File or directory.
const std = @import("std");
const Reader = std.Io.Reader;
const WaitGroup = std.Thread.WaitGroup;
const Pool = std.Thread.Pool;
const Mutex = std.Thread.Mutex;
const Archive = @import("archive.zig");
const DirEntry = @import("dir_entry.zig");
const ExtractionOptions = @import("options.zig");
const dir = @import("inode_data/dir.zig");
const file = @import("inode_data/file.zig");
const misc = @import("inode_data/misc.zig");
const Tables = @import("tables.zig");
const DataReader = @import("util/data.zig");
const ThreadedDataReader = @import("util/data_threaded.zig");
const InodeExtract = @import("util/extract.zig");
const InodeFinish = @import("util/inode_finish.zig");
const FinishUnion = InodeFinish.FinishUnion;
const MetadataReader = @import("util/metadata.zig");
pub const Ref = packed struct {
block_offset: u16,
block_start: u32,
_: u16,
};
pub const InodeType = enum(u16) {
dir = 1,
file,
symlink,
block_dev,
char_dev,
fifo,
socket,
ext_dir,
ext_file,
ext_symlink,
ext_block_dev,
ext_char_dev,
ext_fifo,
ext_socket,
};
pub const InodeData = union(InodeType) {
dir: dir.Dir,
file: file.File,
symlink: misc.Symlink,
block_dev: misc.Dev,
char_dev: misc.Dev,
fifo: misc.IPC,
socket: misc.IPC,
ext_dir: dir.ExtDir,
ext_file: file.ExtFile,
ext_symlink: misc.ExtSymlink,
ext_block_dev: misc.ExtDev,
ext_char_dev: misc.ExtDev,
ext_fifo: misc.ExtIPC,
ext_socket: misc.ExtIPC,
};
pub const Header = packed struct {
inode_type: InodeType,
permissions: u16,
uid_idx: u16,
gid_idx: u16,
mod_time: u32,
num: u32,
};
const Inode = @This();
hdr: Header,
data: InodeData,
pub fn read(alloc: std.mem.Allocator, rdr: *Reader, block_size: u32) !Inode {
var hdr: Header = undefined;
try rdr.readSliceEndian(Header, @ptrCast(&hdr), .little);
return .{
.hdr = hdr,
.data = switch (hdr.inode_type) {
.dir => .{ .dir = try .read(rdr) },
.file => .{ .file = try .read(alloc, rdr, block_size) },
.symlink => .{ .symlink = try .read(alloc, rdr) },
.block_dev => .{ .block_dev = try .read(rdr) },
.char_dev => .{ .char_dev = try .read(rdr) },
.fifo => .{ .fifo = try .read(rdr) },
.socket => .{ .socket = try .read(rdr) },
.ext_dir => .{ .ext_dir = try .read(rdr) },
.ext_file => .{ .ext_file = try .read(alloc, rdr, block_size) },
.ext_symlink => .{ .ext_symlink = try .read(alloc, rdr) },
.ext_block_dev => .{ .ext_block_dev = try .read(rdr) },
.ext_char_dev => .{ .ext_char_dev = try .read(rdr) },
.ext_fifo => .{ .ext_fifo = try .read(rdr) },
.ext_socket => .{ .ext_socket = try .read(rdr) },
},
};
}
pub fn readFromEntry(alloc: std.mem.Allocator, archive: Archive, entry: DirEntry) !Inode {
var rdr = try archive.fil.readerAt(archive.super.inode_start + entry.block_start, &[0]u8{});
var meta: MetadataReader = .init(alloc, &rdr.interface, archive.decomp);
try meta.interface.discardAll(entry.block_offset);
return read(alloc, &meta.interface, archive.super.block_size);
}
pub fn deinit(self: Inode, alloc: std.mem.Allocator) void {
switch (self.data) {
.file => |f| alloc.free(f.block_sizes),
.ext_file => |f| alloc.free(f.block_sizes),
.symlink => |s| alloc.free(s.target),
.ext_symlink => |s| alloc.free(s.target),
else => {},
}
}
/// Get the data reader for a file inode.
pub fn dataReader(self: Inode, alloc: std.mem.Allocator, archive: Archive, tables: *Tables) !DataReader {
return switch (self.hdr.inode_type) {
.file => readerFromData(alloc, archive, tables, self.data.file),
.ext_file => readerFromData(alloc, archive, tables, self.data.ext_file),
else => error.NotRegularFile,
};
}
fn readerFromData(alloc: std.mem.Allocator, archive: Archive, tables: *Tables, data: anytype) !DataReader {
var out: DataReader = .init(alloc, archive, data.block_sizes, data.block_start, data.size);
if (data.frag_idx != 0xFFFFFFFF)
out.addFragment(try tables.frag_table.get(data.frag_idx), data.frag_block_offset);
return out;
}
/// Get the directory entries for a directory inode.
pub fn dirEntries(self: Inode, alloc: std.mem.Allocator, archive: Archive) ![]DirEntry {
return switch (self.hdr.inode_type) {
.dir => entriesFromData(alloc, archive, self.data.dir),
.ext_dir => entriesFromData(alloc, archive, self.data.ext_dir),
else => error.NotDirectory,
};
}
fn entriesFromData(alloc: std.mem.Allocator, archive: Archive, data: anytype) ![]DirEntry {
var rdr = try archive.fil.readerAt(archive.super.dir_start + data.block_start, &[0]u8{});
var meta: MetadataReader = .init(alloc, &rdr.interface, archive.decomp);
try meta.interface.discardAll(data.block_offset);
return DirEntry.readDir(alloc, &meta.interface, data.size);
}
/// Returns the xattr index for the given inode. If the inode isn't an extended variant or doesn't have any, the u32 max is returned (0xFFFFFFFF).
pub fn xattrIdx(self: Inode) u32 {
return switch (self.data) {
.ext_dir => |d| d.xattr_id,
.ext_file => |f| f.xattr_idx,
.ext_symlink => |s| s.xattr_idx,
.ext_block_dev, .ext_char_dev => |d| d.xattr_idx,
.ext_fifo, .ext_socket => |i| i.xattr_idx,
else => 0xFFFFFFFF,
};
}
/// Applies the Inode's metadata to the given File.
/// Mod time is always set, but permissions and xattrs are set based on the given ExtractionOptions.
pub fn setMetadata(self: Inode, alloc: std.mem.Allocator, tables: *Tables, fil: std.fs.File, options: ExtractionOptions) !void {
const time = @as(i128, self.hdr.mod_time) * 1000000000;
try fil.updateTimes(time, time);
if (!options.ignore_permissions) {
try fil.chmod(self.hdr.permissions);
try fil.chown(try tables.id_table.get(self.hdr.uid_idx), try tables.id_table.get(self.hdr.gid_idx));
}
if (!options.ignore_xattr) {
const idx = self.xattrIdx();
if (idx == 0xFFFFFFFF) return;
const xattrs = try tables.xattr_table.get(alloc, idx);
defer alloc.free(xattrs);
for (xattrs) |kv| {
const res = std.os.linux.fsetxattr(fil.handle, kv.key, kv.value.ptr, kv.value.len, 0);
alloc.free(kv.key);
alloc.free(kv.value);
if (res != 0) {
if (options.verbose)
options.verbose_writer.?.print("fsetxattr has result of: {}\n", .{res}) catch {};
return error.SetXattr;
}
}
}
}
/// Extract the inode to the given path.
pub fn extractTo(self: Inode, alloc: std.mem.Allocator, archive: Archive, path: []const u8, options: ExtractionOptions) !void {
return InodeExtract.extractTo(alloc, self, archive, path, options);
}
+32
View File
@@ -0,0 +1,32 @@
const Reader = @import("std").Io.Reader;
pub const Dir = packed struct {
block_start: u32,
hard_links: u32,
size: u16,
block_offset: u16,
parent_num: u32,
pub fn read(rdr: *Reader) !Dir {
var d: Dir = undefined;
try rdr.readSliceEndian(Dir, @ptrCast(&d), .little);
return d;
}
};
pub const ExtDir = packed struct {
hard_links: u32,
size: u32,
block_start: u32,
parent_num: u32,
idx_count: u16,
block_offset: u16,
xattr_id: u32,
// index: []DirIndex
pub fn read(rdr: *Reader) !ExtDir {
var d: ExtDir = undefined;
try rdr.readSliceEndian(Dir, @ptrCast(&d), .little);
return d;
}
};
+76
View File
@@ -0,0 +1,76 @@
const std = @import("std");
const Reader = std.Io.Reader;
pub const BlockSize = packed struct {
size: u24,
uncompressed: bool,
_: u7,
};
pub const File = struct {
block_start: u32, // bytes 0-3
frag_idx: u32, // bytes 4-7
frag_block_offset: u32, // bytes 8-11
size: u32, // bytes 12-15
block_sizes: []BlockSize,
pub fn read(alloc: std.mem.Allocator, rdr: *Reader, block_size: u32) !File {
var start: [16]u8 = undefined;
try rdr.readSliceAll(&start);
const frag_idx: u32 = std.mem.readInt(u32, start[4..8], .little);
const size: u32 = std.mem.readInt(u32, start[12..16], .little);
var num_blocks: u32 = size / block_size;
if (size % block_size != 0 and frag_idx == 0xFFFFFFFF) num_blocks += 1;
const sizes = try alloc.alloc(BlockSize, num_blocks);
errdefer alloc.free(sizes);
try rdr.readSliceEndian(BlockSize, sizes, .little);
return .{
.block_start = std.mem.readInt(u32, start[0..4], .little),
.frag_idx = frag_idx,
.frag_block_offset = std.mem.readInt(u32, start[8..12], .little),
.size = size,
.block_sizes = sizes,
};
}
pub fn deinit(self: File, alloc: std.mem.Allocator) void {
alloc.free(self.block_sizes);
}
};
pub const ExtFile = struct {
block_start: u64, // bytes 0-7
size: u64, // bytes 8-15
sparse: u64, // bytes 16-23
hard_links: u32, // bytes 24-27
frag_idx: u32, // bytes 28-31
frag_block_offset: u32, // bytes 32-35
xattr_idx: u32, // bytes 36-39
block_sizes: []BlockSize,
pub fn read(alloc: std.mem.Allocator, rdr: *Reader, block_size: u32) !ExtFile {
var start: [40]u8 = undefined;
try rdr.readSliceAll(&start);
const frag_idx: u32 = std.mem.readInt(u32, start[28..32], .little);
const size: u64 = std.mem.readInt(u64, start[8..16], .little);
var num_blocks: u32 = @truncate(size / block_size);
if (size % block_size != 0 and frag_idx == 0xFFFFFFFF) num_blocks += 1;
const sizes = try alloc.alloc(BlockSize, num_blocks);
errdefer alloc.free(sizes);
try rdr.readSliceEndian(BlockSize, sizes, .little);
return .{
.block_start = std.mem.readInt(u64, start[0..8], .little),
.size = size,
.sparse = std.mem.readInt(u64, start[16..24], .little),
.hard_links = std.mem.readInt(u32, start[24..28], .little),
.frag_idx = frag_idx,
.frag_block_offset = std.mem.readInt(u32, start[32..36], .little),
.xattr_idx = std.mem.readInt(u32, start[36..40], .little),
.block_sizes = sizes,
};
}
pub fn deinit(self: ExtFile, alloc: std.mem.Allocator) void {
alloc.free(self.block_sizes);
}
};
+98
View File
@@ -0,0 +1,98 @@
const std = @import("std");
const Reader = std.Io.Reader;
pub const Symlink = struct {
hard_links: u32,
target: []const u8,
pub fn read(alloc: std.mem.Allocator, rdr: *Reader) !Symlink {
var start: [8]u8 = undefined;
try rdr.readSliceAll(&start);
const target_size = std.mem.readInt(u32, start[4..8], .little);
const target = try alloc.alloc(u8, target_size + 1);
errdefer alloc.free(target);
try rdr.readSliceEndian(u8, target, .little);
return .{
.hard_links = std.mem.readInt(u32, start[0..4], .little),
.target = target,
};
}
pub fn deinit(self: Symlink, alloc: std.mem.Allocator) void {
alloc.free(self.target);
}
};
pub const ExtSymlink = struct {
hard_links: u32,
target: []const u8,
xattr_idx: u32,
pub fn read(alloc: std.mem.Allocator, rdr: *Reader) !ExtSymlink {
var start: [8]u8 = undefined;
try rdr.readSliceAll(&start);
const target_size = std.mem.readInt(u32, start[4..8], .little);
const target = try alloc.alloc(u8, target_size + 1);
errdefer alloc.free(target);
try rdr.readSliceEndian(u8, target, .little);
var xattr_idx: u32 = undefined;
try rdr.readSliceEndian(u32, @ptrCast(&xattr_idx), .little);
return .{
.hard_links = std.mem.readInt(u32, start[0..4], .little),
.target = target,
.xattr_idx = xattr_idx,
};
}
pub fn deinit(self: ExtSymlink, alloc: std.mem.Allocator) void {
alloc.free(self.target);
}
};
/// A block or character device.
pub const Dev = packed struct {
hard_links: u32,
dev: u32,
pub fn read(rdr: *Reader) !Dev {
var d: Dev = undefined;
try rdr.readSliceEndian(Dev, @ptrCast(&d), .little);
return d;
}
};
/// An extended block or character device.
pub const ExtDev = packed struct {
hard_links: u32,
dev: u32,
xattr_idx: u32,
pub fn read(rdr: *Reader) !ExtDev {
var d: ExtDev = undefined;
try rdr.readSliceEndian(ExtDev, @ptrCast(&d), .little);
return d;
}
};
/// A socket or FIFO file.
pub const IPC = packed struct {
hard_links: u32,
pub fn read(rdr: *Reader) !IPC {
var d: IPC = undefined;
try rdr.readSliceEndian(IPC, @ptrCast(&d), .little);
return d;
}
};
/// An extended socket or FIFO file.
pub const ExtIPC = packed struct {
hard_links: u32,
xattr_idx: u32,
pub fn read(rdr: *Reader) !ExtIPC {
var d: ExtIPC = undefined;
try rdr.readSliceEndian(ExtIPC, @ptrCast(&d), .little);
return d;
}
};
+33
View File
@@ -0,0 +1,33 @@
//! Options for file/directory extraction.
const std = @import("std");
const Writer = std.Io.Writer;
const ExtractionOptions = @This();
/// The number of threads used for extraction. 0 implies single threaded.
threads: usize = 1,
/// Don't set the file's owner & permissions after extraction
ignore_permissions: bool = false,
/// Don't set xattr values. Currently xattrs are never set anyway.
ignore_xattr: bool = false,
/// Replace symlinks with their target.
dereference_symlinks: bool = false,
/// Verbose logging. If true, verbose_writer must be set
verbose: bool = false,
/// Where to print verbose log.
verbose_writer: ?*Writer = null,
pub const SingleThreadedDefault: ExtractionOptions = .{};
pub fn Default() !ExtractionOptions {
return .{
.threads = try std.Thread.getCpuCount(),
};
}
pub fn VerboseDefault(wrt: *Writer) !ExtractionOptions {
return .{
.verbose = true,
.verbose_writer = wrt,
.threads = try std.Thread.getCpuCount(),
};
}
+2
View File
@@ -0,0 +1,2 @@
pub const Archive = @import("archive.zig");
pub const ExtractionOptions = @import("options.zig");
+63
View File
@@ -0,0 +1,63 @@
const std = @import("std");
const math = std.math;
const CompressionType = @import("decomp.zig").CompressionType;
const InodeRef = @import("inode.zig").Ref;
const SQUASHFS_MAGIC: u32 = std.mem.readInt(u32, "hsqs", .little);
const SuperblockError = error{
InvalidMagic,
InvalidBlockLog,
InvalidVersion,
InvalidCheck,
};
/// A squashfs Superblock
pub const Superblock = packed struct {
magic: u32,
inode_count: u32,
mod_time: u32,
block_size: u32,
frag_count: u32,
compression: CompressionType,
block_log: u16,
flags: packed struct {
inode_uncompressed: bool,
data_uncompressed: bool,
check: bool,
frag_uncompressed: bool,
fragment_never: bool,
fragment_always: bool,
duplicates: bool,
exportable: bool,
xattr_uncompressed: bool,
xattr_never: bool,
compression_options: bool,
ids_uncompressed: bool,
_: u4,
},
id_count: u16,
ver_maj: u16,
ver_min: u16,
root_ref: InodeRef,
size: u64,
id_start: u64,
xattr_start: u64,
inode_start: u64,
dir_start: u64,
frag_start: u64,
export_start: u64,
/// Validate the Superblock. If an error is returned, it's likely the archive is corrupted or not a squashfs archive.
pub fn validate(self: Superblock) !void {
if (self.magic != SQUASHFS_MAGIC)
return SuperblockError.InvalidMagic;
if (self.flags.check)
return SuperblockError.InvalidCheck;
if (self.ver_maj != 4 or self.ver_min != 0)
return SuperblockError.InvalidVersion;
if (math.log2(self.block_size) != self.block_log)
return SuperblockError.InvalidBlockLog;
}
};
+108
View File
@@ -0,0 +1,108 @@
const std = @import("std");
const Mutex = std.Thread.Mutex;
const Archive = @import("archive.zig");
const DecompFn = @import("decomp.zig").DecompFn;
const BlockSize = @import("inode_data/file.zig").BlockSize;
const InodeRef = @import("inode.zig").Ref;
const Superblock = @import("super.zig").Superblock;
const MetadataReader = @import("util/metadata.zig");
const OffsetFile = @import("util/offset_file.zig");
const XattrTable = @import("xattr.zig");
/// Information about a fragment section. Multiple fragments are contained in the block described by a single FragEntry.
/// The offset into the block and fragment size is stored in the file's inode.
pub const FragEntry = packed struct {
start: u64,
size: BlockSize,
_: u32,
};
const Tables = @This();
frag_table: Table(FragEntry),
id_table: Table(u16),
export_table: Table(InodeRef),
xattr_table: XattrTable,
pub fn init(alloc: std.mem.Allocator, archive: Archive) !Tables {
return .{
.frag_table = try .init(alloc, archive.fil, archive.decomp, archive.super.frag_start, archive.super.frag_count),
.id_table = try .init(alloc, archive.fil, archive.decomp, archive.super.id_start, archive.super.id_count),
.export_table = try .init(alloc, archive.fil, archive.decomp, archive.super.export_start, archive.super.inode_count),
.xattr_table = try .init(alloc, archive.fil, archive.decomp, archive.super.xattr_start),
};
}
pub fn deinit(self: *Tables) void {
self.frag_table.deinit();
self.id_table.deinit();
self.export_table.deinit();
self.xattr_table.deinit();
}
/// A two-layer metadata table.
pub fn Table(T: anytype) type {
return struct {
const Self = @This();
const VALS_PER_BLOCK = 8192 / @sizeOf(T);
alloc: std.mem.Allocator,
fil: OffsetFile,
decomp: DecompFn,
tab_start: u64,
tab: std.AutoHashMap(u32, []T),
values: u32,
mut: Mutex = .{},
pub fn init(alloc: std.mem.Allocator, fil: OffsetFile, decomp: DecompFn, tab_start: u64, values: u32) !Self {
return .{
.alloc = alloc,
.fil = fil,
.decomp = decomp,
.tab_start = tab_start,
.tab = .init(alloc),
.values = values,
};
}
pub fn deinit(self: *Self) void {
var iter = self.tab.valueIterator();
while (iter.next()) |s| {
self.alloc.free(s.*);
}
self.tab.deinit();
}
pub fn get(self: *Self, idx: u32) !T {
if (idx >= self.values) return error.InvalidIndex;
const block_num = idx / VALS_PER_BLOCK;
const idx_offset = idx - (block_num * VALS_PER_BLOCK);
if (self.tab.contains(block_num)) {
const block = self.tab.get(block_num).?;
return block[idx_offset];
}
self.mut.lock();
defer self.mut.unlock();
// Double check in case of race condition..
if (self.tab.contains(block_num)) {
const block = self.tab.get(block_num).?;
return block[idx_offset];
}
const is_last = (self.values - 1) / VALS_PER_BLOCK == block_num;
const slice_size = if (is_last) self.values - (block_num * VALS_PER_BLOCK) else VALS_PER_BLOCK;
const slice = try self.alloc.alloc(T, slice_size);
var rdr = try self.fil.readerAt(self.tab_start + (8 * block_num), &[0]u8{});
var offset: u64 = 0;
try rdr.interface.readSliceEndian(u64, @ptrCast(&offset), .little);
rdr = try self.fil.readerAt(offset, &[0]u8{});
var meta: MetadataReader = .init(self.alloc, &rdr.interface, self.decomp);
try meta.interface.readSliceEndian(T, @ptrCast(slice), .little);
try self.tab.put(block_num, slice);
return slice[idx_offset];
}
};
}
+84
View File
@@ -0,0 +1,84 @@
const std = @import("std");
const stuff = @import("builtin");
const Archive = @import("archive.zig");
const Superblock = @import("super.zig").Superblock;
const TestArchive = "testing/LinuxPATest.sfs";
test "Basics" {
var fil = try std.fs.cwd().openFile(TestArchive, .{});
defer fil.close();
var sfs: Archive = try .init(std.testing.allocator, fil);
defer sfs.deinit();
if (sfs.super != LinuxPATestCorrectSuperblock) {
std.debug.print("Superblock wrong\nShould be: {}\n\nis: {}\n", .{ LinuxPATestCorrectSuperblock, sfs.super });
return error.BadSuperblock;
}
}
const TestFile = "Start.exe";
const TestFileExtractLocation = "testing/Start.exe";
test "ExtractSingleFile" {
std.fs.cwd().deleteFile(TestFileExtractLocation) catch {};
var fil = try std.fs.cwd().openFile(TestArchive, .{});
defer fil.close();
var sfs: Archive = try .init(std.testing.allocator, fil);
defer sfs.deinit();
var test_fil = try sfs.open(TestFile);
defer test_fil.deinit();
try test_fil.extract(TestFileExtractLocation, .Default);
//TODO: validate extracted file.
}
const TestFullExtractLocation = "testing/TestExtract";
test "ExtractCompleteArchive" {
std.fs.cwd().deleteTree(TestFullExtractLocation) catch {};
var fil = try std.fs.cwd().openFile(TestArchive, .{});
defer fil.close();
var sfs: Archive = try .init(std.testing.allocator, fil);
defer sfs.deinit();
try sfs.extract(TestFullExtractLocation, .Default);
}
const LinuxPATestCorrectSuperblock: Superblock = .{
.magic = std.mem.readInt(u32, "hsqs", .little),
.inode_count = 2974,
.mod_time = 1632696724,
.block_size = 131072,
.frag_count = 264,
.compression = .zstd,
.block_log = 17,
.flags = .{
.inode_uncompressed = false,
.data_uncompressed = false,
.check = false,
.frag_uncompressed = false,
.fragment_never = false,
.fragment_always = false,
.duplicates = true,
.exportable = true,
.xattr_uncompressed = false,
.xattr_never = false,
.compression_options = false,
.ids_uncompressed = false,
._ = 0,
},
.id_count = 1,
.ver_maj = 4,
.ver_min = 0,
.root_ref = .{
.block_offset = 1363,
.block_start = 29237,
._ = 0,
},
.size = 106841744,
.id_start = 106841632,
.xattr_start = 106841720,
.inode_start = 106778274,
.dir_start = 106807998,
.frag_start = 106837747,
.export_start = 106841602,
};
+174
View File
@@ -0,0 +1,174 @@
//! A reader for a regular file.
const std = @import("std");
const Reader = std.Io.Reader;
const Writer = std.Io.Writer;
const Limit = std.Io.Limit;
const Archive = @import("../archive.zig");
const DecompFn = @import("../decomp.zig").DecompFn;
const BlockSize = @import("../inode_data/file.zig").BlockSize;
const FragEntry = @import("../tables.zig").FragEntry;
const OffsetFile = @import("offset_file.zig");
const DataReader = @This();
alloc: std.mem.Allocator,
fil: OffsetFile,
decomp: DecompFn,
block_size: u32,
blocks: []BlockSize,
frag: ?FragEntry = null, // TODO: do something better?
frag_offset: u32 = 0,
size: u64,
interface: Reader,
cur_offset: u64,
block_idx: u32 = 0,
pub fn init(alloc: std.mem.Allocator, archive: Archive, blocks: []BlockSize, start: u64, size: u64) DataReader {
return .{
.alloc = alloc,
.fil = archive.fil,
.decomp = archive.decomp,
.block_size = archive.super.block_size,
.blocks = blocks,
.size = size,
.cur_offset = start,
.interface = .{
.end = 0,
.seek = 0,
.buffer = &[0]u8{},
.vtable = &.{
.stream = stream,
.discard = discard,
.readVec = readVec,
},
},
};
}
pub fn deinit(self: *DataReader) void {
self.alloc.free(self.interface.buffer);
self.interface.end = 0;
self.interface.seek = 0;
}
pub fn addFragment(self: *DataReader, entry: FragEntry, frag_offset: u32) void {
self.frag = entry;
self.frag_offset = frag_offset;
}
fn numBlocks(self: DataReader) usize {
var res = self.blocks.len;
if (self.frag != null) res += 1;
return res;
}
fn advance(self: *DataReader) !void {
if (self.block_idx > self.blocks.len or (self.block_idx == self.blocks.len and self.frag == null)) {
if (self.interface.buffer.len > 0) {
self.alloc.free(self.interface.buffer);
self.interface.buffer = &[0]u8{};
self.interface.end = 0;
self.interface.seek = 0;
}
return Reader.Error.EndOfStream;
}
defer self.block_idx += 1;
const cur_block_size = if (self.block_idx == self.numBlocks() - 1) self.size % self.block_size else self.block_size;
try self.resizeBuffer(cur_block_size);
self.interface.seek = 0;
self.interface.end = cur_block_size;
if (self.block_idx == self.blocks.len) { // fragment
var rdr = try self.fil.readerAt(self.frag.?.start, &[0]u8{});
if (self.frag.?.size.uncompressed) {
try rdr.interface.discardAll(self.frag_offset);
try rdr.interface.readSliceAll(self.interface.buffer);
return;
}
const tmp_buf = try self.alloc.alloc(u8, self.frag.?.size.size);
defer self.alloc.free(tmp_buf);
try rdr.interface.readSliceAll(tmp_buf);
const needed_block = try self.alloc.alloc(u8, self.block_size);
defer self.alloc.free(needed_block);
_ = try self.decomp(self.alloc, tmp_buf, needed_block);
@memcpy(self.interface.buffer, needed_block[self.frag_offset .. self.frag_offset + cur_block_size]);
return;
}
const block = self.blocks[self.block_idx];
if (block.size == 0) {
@memset(self.interface.buffer, 0);
return;
}
var rdr = try self.fil.readerAt(self.cur_offset, &[0]u8{});
self.cur_offset += block.size;
if (block.uncompressed) {
try rdr.interface.readSliceAll(self.interface.buffer);
return;
}
const tmp_buf = try self.alloc.alloc(u8, block.size);
defer self.alloc.free(tmp_buf);
try rdr.interface.readSliceAll(tmp_buf);
_ = try self.decomp(self.alloc, tmp_buf, self.interface.buffer);
}
/// Does not guarentee that data currently in the buffer is retained.
fn resizeBuffer(self: *DataReader, size: usize) !void {
if (self.interface.buffer.len == size) return;
if (!self.alloc.resize(self.interface.buffer, size)) {
self.alloc.free(self.interface.buffer);
self.interface.buffer = self.alloc.alloc(u8, size) catch |err| {
self.interface.buffer = &[0]u8{};
return err;
};
} else {
self.interface.buffer.len = size;
}
}
fn stream(rdr: *Reader, wrt: *Writer, limit: Limit) Reader.StreamError!usize {
var self: *DataReader = @alignCast(@fieldParentPtr("interface", rdr));
if (rdr.seek >= rdr.end) self.advance() catch |err| {
if (err == error.EndOfStream) return error.EndOfStream;
std.log.err("Error advancing data reader: {}\n", .{err});
return Reader.Error.ReadFailed;
};
if (limit == .nothing) return 0;
const to_read = @min(rdr.end - rdr.seek, @intFromEnum(limit));
const res = try wrt.write(rdr.buffer[rdr.seek .. rdr.seek + to_read]);
rdr.seek += res;
return res;
}
fn discard(rdr: *Reader, limit: Limit) Reader.Error!usize {
var self: *DataReader = @alignCast(@fieldParentPtr("interface", rdr));
if (rdr.seek >= rdr.end) self.advance() catch |err| {
if (err == error.EndOfStream) return error.EndOfStream;
std.log.err("Error advancing data reader: {}\n", .{err});
return Reader.Error.ReadFailed;
};
if (limit == .nothing) return 0;
const to_adv = @min(rdr.end - rdr.seek, @intFromEnum(limit));
rdr.seek += to_adv;
return to_adv;
}
fn readVec(rdr: *Reader, vec: [][]u8) Reader.Error!usize {
var self: *DataReader = @alignCast(@fieldParentPtr("interface", rdr));
if (rdr.seek >= rdr.end) self.advance() catch |err| {
if (err == error.EndOfStream) return error.EndOfStream;
std.log.err("Error advancing data reader: {}\n", .{err});
return Reader.Error.ReadFailed;
};
var cur_red: usize = 0;
for (vec) |s| {
const to_copy: usize = @min(rdr.end - rdr.seek, s.len);
@memcpy(s[0..to_copy], rdr.buffer[rdr.seek .. rdr.seek + to_copy]);
rdr.seek += to_copy;
cur_red += to_copy;
if (rdr.end == rdr.seek) break;
}
return cur_red;
}
+207
View File
@@ -0,0 +1,207 @@
//! Similiar to DataReader, but set-up for threaded writing to files.
const std = @import("std");
const Reader = std.Io.Reader;
const Writer = std.Io.Writer;
const Limit = std.Io.Limit;
const WaitGroup = std.Thread.WaitGroup;
const Pool = std.Thread.Pool;
const Archive = @import("../archive.zig");
const DecompFn = @import("../decomp.zig").DecompFn;
const BlockSize = @import("../inode_data/file.zig").BlockSize;
const FragEntry = @import("../tables.zig").FragEntry;
const InodeFinish = @import("inode_finish.zig");
const OffsetFile = @import("offset_file.zig");
const ThreadedDataReader = @This();
alloc: std.mem.Allocator,
fil: OffsetFile,
decomp: DecompFn,
block_size: u32,
blocks: []BlockSize,
frag: ?FragEntry = null, // TODO: do something better?
frag_offset: u32 = 0,
size: u64,
num_blocks: usize,
start_offset: u64,
pub fn init(alloc: std.mem.Allocator, archive: Archive, blocks: []BlockSize, start: u64, size: u64) ThreadedDataReader {
return .{
.alloc = alloc,
.fil = archive.fil,
.decomp = archive.decomp,
.block_size = archive.super.block_size,
.blocks = blocks,
.size = size,
.num_blocks = blocks.len,
.start_offset = start,
};
}
pub fn addFragment(self: *ThreadedDataReader, entry: FragEntry, frag_offset: u32) void {
self.frag = entry;
self.frag_offset = frag_offset;
self.num_blocks = self.blocks.len + 1;
}
/// Extract the data to the file threadedly, using pool to spawn threads.
/// If errors occur, they are set to finish.out_err.
pub fn extractThreaded(self: ThreadedDataReader, file: std.fs.File, pool: *Pool, finish: *InodeFinish) void {
var cur_write_offset: u64 = 0;
var cur_read_offset: u64 = self.start_offset;
for (0..self.blocks.len) |i| {
const cur_block_size = if (i == self.num_blocks - 1) self.size % self.block_size else self.block_size;
pool.spawn(workThreadBlocks, .{ self, file, cur_write_offset, cur_read_offset, self.blocks[i], cur_block_size, finish }) catch |res_err| {
finish.logError("Can't spawn pool task: {}", .{res_err});
finish.out_err.* = res_err;
finish.finish();
};
cur_write_offset += cur_block_size;
cur_read_offset += self.blocks[i].size;
}
if (self.frag != null)
pool.spawn(workThreadFragment, .{ self, file, cur_write_offset, finish }) catch |res_err| {
finish.logError("Can't spawn pool task: {}", .{res_err});
finish.out_err.* = res_err;
finish.finish();
};
}
fn workThreadBlocks(
self: ThreadedDataReader,
fil: std.fs.File,
write_offset: u64,
read_offset: u64,
block: BlockSize,
cur_block_size: u64,
finish: *InodeFinish,
) void {
defer finish.finish();
var wrt = fil.writer(&[0]u8{});
wrt.seekTo(write_offset) catch |err| {
finish.logError("Error seeking file writer: {}", .{err});
finish.out_err.* = err;
return;
};
defer wrt.interface.flush() catch |err| {
finish.logError("Error flushing file writer: {}", .{err});
finish.out_err.* = err;
};
if (block.size == 0) {
wrt.interface.splatByteAll(0, cur_block_size) catch |err| {
finish.logError("Error writing zeroes: {}", .{err});
finish.out_err.* = err;
return;
};
return;
}
var rdr = self.fil.readerAt(read_offset, &[0]u8{}) catch |err| {
finish.logError("Error creating file reader: {}", .{err});
finish.out_err.* = err;
return;
};
if (block.uncompressed) {
rdr.interface.streamExact(&wrt.interface, block.size) catch |err| {
finish.logError("Error streaming data: {}", .{err});
finish.out_err.* = err;
return;
};
return;
}
// TODO: shared buffers
const read_buf = self.alloc.alloc(u8, block.size) catch |err| {
finish.logError("Error creating reader buffer: {}", .{err});
finish.out_err.* = err;
return;
};
defer self.alloc.free(read_buf);
rdr.interface.readSliceAll(read_buf) catch |err| {
finish.logError("Error reading data into reader buffer: {}", .{err});
finish.out_err.* = err;
return;
};
// TODO: shared buffers
const res_buf = self.alloc.alloc(u8, cur_block_size) catch |err| {
finish.logError("Error creating result buffer: {}", .{err});
finish.out_err.* = err;
return;
};
defer self.alloc.free(res_buf);
_ = self.decomp(self.alloc, read_buf, res_buf) catch |err| {
finish.logError("Error decompressing data block: {}", .{err});
finish.out_err.* = err;
return;
};
wrt.interface.writeAll(res_buf) catch |err| {
finish.logError("Error writing to file: {}", .{err});
finish.out_err.* = err;
return;
};
}
fn workThreadFragment(self: ThreadedDataReader, fil: std.fs.File, write_offset: u64, finish: *InodeFinish) void {
defer finish.finish();
var wrt = fil.writer(&[0]u8{});
wrt.seekTo(write_offset) catch |err| {
finish.logError("Error seeking file writer for file fragment: {}", .{err});
finish.out_err.* = err;
return;
};
defer wrt.interface.flush() catch |err| {
finish.out_err.* = err;
};
var rdr = self.fil.readerAt(self.frag.?.start, &[0]u8{}) catch |err| {
finish.logError("Error creating file reader for file fragment: {}", .{err});
finish.out_err.* = err;
return;
};
if (self.frag.?.size.uncompressed) {
rdr.interface.discardAll(self.frag_offset) catch |err| {
finish.logError("Error discarding useless fragment data: {}", .{err});
finish.out_err.* = err;
return;
};
rdr.interface.streamExact(&wrt.interface, self.size % self.block_size) catch |err| {
finish.logError("Error streaming fragment data: {}", .{err});
finish.out_err.* = err;
return;
};
return;
}
const tmp_buf = self.alloc.alloc(u8, self.frag.?.size.size) catch |err| {
finish.logError("Error creating a temporary buffer for a file fragment: {}", .{err});
finish.out_err.* = err;
return;
};
defer self.alloc.free(tmp_buf);
rdr.interface.readSliceAll(tmp_buf) catch |err| {
finish.logError("Error reading data into fragment buffer: {}", .{err});
finish.out_err.* = err;
return;
};
const needed_block = self.alloc.alloc(u8, self.block_size) catch |err| {
finish.logError("Error allocating fragment decompression results: {}", .{err});
finish.out_err.* = err;
return;
};
defer self.alloc.free(needed_block);
_ = self.decomp(self.alloc, tmp_buf, needed_block) catch |err| {
finish.logError("Error decompressing fragment: {}", .{err});
finish.out_err.* = err;
return;
};
wrt.interface.writeAll(needed_block[self.frag_offset .. self.frag_offset + (self.size % self.block_size)]) catch |err| {
finish.logError("Error writing fragment: {}", .{err});
finish.out_err.* = err;
return;
};
}
+353
View File
@@ -0,0 +1,353 @@
const std = @import("std");
const Allocator = std.mem.Allocator;
const Pool = std.Thread.Pool;
const WaitGroup = std.Thread.WaitGroup;
const Archive = @import("../archive.zig");
const DirEntry = @import("../dir_entry.zig");
const Inode = @import("../inode.zig");
const ExtractionOptions = @import("../options.zig");
const Tables = @import("../tables.zig");
const InodeFinish = @import("inode_finish.zig");
const FinishUnion = InodeFinish.FinishUnion;
const ThreadedDataReader = @import("data_threaded.zig");
// 1 MB
const STACK_ALLOC_SIZE = 1024 * 1024;
pub fn extractTo(
allocator: Allocator,
inode: Inode,
archive: Archive,
path: []const u8,
options: ExtractionOptions,
) !void {
if (path[path.len - 1] == '/')
return extractTo(allocator, inode, archive, path[0 .. path.len - 2], options);
var stack_alloc = std.heap.stackFallback(STACK_ALLOC_SIZE, allocator);
var arena: std.heap.ArenaAllocator = .init(stack_alloc.get());
defer arena.deinit();
if (options.threads <= 1) {
const alloc = arena.allocator();
var tables: Tables = try .init(alloc, archive);
return extractSingleThread(arena.allocator(), inode, archive, &tables, path, options);
}
var thread_alloc = std.heap.ThreadSafeAllocator{ .child_allocator = arena.allocator() };
const alloc = thread_alloc.allocator();
var tables: Tables = try .init(alloc, archive);
var pool_alloc = std.heap.stackFallback(10 * 1024, alloc);
var pool: Pool = undefined;
try pool.init(.{ .allocator = pool_alloc.get(), .n_jobs = options.threads - 1 });
var wg: WaitGroup = .{};
var err: ?anyerror = null;
wg.start();
try pool.spawn(extractMultiThread, .{
alloc,
inode,
archive,
&tables,
path,
options,
&pool,
FinishUnion{ .wg = &wg },
&err,
});
pool.waitAndWork(&wg);
if (err != null) return err.?;
}
fn extractSingleThread(
alloc: Allocator,
inode: Inode,
archive: Archive,
tables: *Tables,
path: []const u8,
options: ExtractionOptions,
) !void {
switch (inode.hdr.inode_type) {
.dir, .ext_dir => {
_ = std.fs.cwd().makeDir(path) catch |err| switch (err) {
std.fs.Dir.MakeError.PathAlreadyExists => {},
else => return err,
};
// Currently we are ignoring any deinit or free calls since we know we are under an ArenaAllocator.
// Possibly in the future, do some simple math to see if it would be safe to ONLY deinit via Arena,
// otherwise be more conscientious about freeing memory.
// For now, this is good enough.
const entries = try inode.dirEntries(alloc, archive);
for (entries) |ent| {
const sub_inode: Inode = try .readFromEntry(alloc, archive, ent);
const new_path = try std.mem.concat(alloc, u8, &[_][]const u8{ path, "/", ent.name });
try extractSingleThread(alloc, sub_inode, archive, tables, new_path, options);
}
const fil = try std.fs.cwd().openFile(path, .{});
defer fil.close();
try inode.setMetadata(alloc, tables, fil, options);
},
.file, .ext_file => {
var fil = try std.fs.cwd().createFile(path, .{ .exclusive = true });
defer fil.close();
var wrt = fil.writer(&[0]u8{});
var dat_rdr = try inode.dataReader(alloc, archive, tables);
defer dat_rdr.deinit();
_ = try dat_rdr.interface.streamRemaining(&wrt.interface);
try wrt.interface.flush();
try inode.setMetadata(alloc, tables, fil, options);
},
.symlink, .ext_symlink => return extractSymlink(inode, path),
else => return extractDeviceAndIPC(inode, alloc, tables, path, options),
}
}
fn extractMultiThread(
alloc: Allocator,
inode: Inode,
archive: Archive,
tables: *Tables,
path: []const u8,
options: ExtractionOptions,
pool: *Pool,
fin: FinishUnion,
err: *?anyerror,
) void {
if (err.* != null) {
fin.finish();
return;
}
switch (inode.hdr.inode_type) {
.dir, .ext_dir => {
_ = std.fs.cwd().makeDir(path) catch |res_err| switch (res_err) {
std.fs.Dir.MakeError.PathAlreadyExists => {},
else => {
err.* = res_err;
fin.finish();
return;
},
};
// Currently we are ignoring any deinit or free calls since we know we are under an ArenaAllocator.
// Possibly in the future, do some simple math to see if it would be safe to ONLY deinit via Arena,
// otherwise be more conscientious about freeing memory.
// For now, this is good enough.
const entries = inode.dirEntries(alloc, archive) catch |res_err| {
err.* = res_err;
fin.finish();
return;
};
if (entries.len == 0) {
fin.finish();
return;
}
var dir_fin = InodeFinish.create(
alloc,
inode,
path,
tables,
options,
fin,
err,
null,
entries.len,
) catch |res_err| {
err.* = res_err;
fin.finish();
return;
};
for (entries) |ent| {
if (ent.inode_type == .dir) {
extractEntry(
alloc,
ent,
archive,
tables,
path,
options,
pool,
.{ .fin = dir_fin },
err,
);
continue;
}
pool.spawn(
extractEntry,
.{ alloc, ent, archive, tables, path, options, pool, FinishUnion{ .fin = dir_fin }, err },
) catch |res_err| {
err.* = res_err;
dir_fin.finish();
return;
};
}
},
.file, .ext_file => {
const fil = std.fs.cwd().createFile(path, .{ .exclusive = true }) catch |res_err| {
if (options.verbose)
options.verbose_writer.?.print("Can't create file at {s}: {}\n", .{ path, res_err }) catch {};
err.* = res_err;
fin.finish();
return;
};
var data_rdr = threadedDataReader(inode, alloc, archive, tables) catch |res_err| {
if (options.verbose)
options.verbose_writer.?.print("Can't create data reader for inode #{} (extracting to {s}): {}\n", .{ inode.hdr.num, path, res_err }) catch {};
err.* = res_err;
fin.finish();
return;
};
if (data_rdr == null) {
inode.setMetadata(alloc, tables, fil, options) catch |res_err| {
if (options.verbose)
options.verbose_writer.?.print("Can't set metadata to {s}: {}\n", .{ path, res_err }) catch {};
err.* = res_err;
};
fin.finish();
return;
}
const file_fin = InodeFinish.create(
alloc,
inode,
path,
tables,
options,
fin,
err,
fil,
data_rdr.?.num_blocks,
) catch |res_err| {
if (options.verbose)
options.verbose_writer.?.print("Can't create callback for inode #{} (extracting to {s}): {}\n", .{ inode.hdr.num, path, res_err }) catch {};
err.* = res_err;
fin.finish();
return;
};
data_rdr.?.extractThreaded(fil, pool, file_fin);
},
.symlink, .ext_symlink => {
extractSymlink(inode, path) catch |res_err| {
err.* = res_err;
};
fin.finish();
},
else => {
extractDeviceAndIPC(inode, alloc, tables, path, options) catch |res_err| {
err.* = res_err;
};
fin.finish();
},
}
}
fn extractEntry(
alloc: Allocator,
ent: DirEntry,
archive: Archive,
tables: *Tables,
path: []const u8,
options: ExtractionOptions,
pool: *Pool,
fin: FinishUnion,
err: *?anyerror,
) void {
const new_path = std.mem.concat(alloc, u8, &[_][]const u8{ path, "/", ent.name }) catch |res_err| {
err.* = res_err;
fin.finish();
return;
};
const inode = Inode.readFromEntry(alloc, archive, ent) catch |res_err| {
err.* = res_err;
fin.finish();
return;
};
extractMultiThread(alloc, inode, archive, tables, new_path, options, pool, fin, err);
}
/// Get a threaded data reader for a file inode.
fn threadedDataReader(self: Inode, alloc: std.mem.Allocator, archive: Archive, tables: *Tables) !?ThreadedDataReader {
return switch (self.hdr.inode_type) {
.file => threadedReaderFromData(alloc, archive, tables, self.data.file),
.ext_file => threadedReaderFromData(alloc, archive, tables, self.data.ext_file),
else => error.NotRegularFile,
};
}
fn threadedReaderFromData(alloc: std.mem.Allocator, archive: Archive, tables: *Tables, data: anytype) !?ThreadedDataReader {
if (data.block_sizes.len == 0 and data.frag_idx == 0xFFFFFFFF) return null;
var out: ThreadedDataReader = .init(alloc, archive, data.block_sizes, data.block_start, data.size);
if (data.frag_idx != 0xFFFFFFFF)
out.addFragment(try tables.frag_table.get(data.frag_idx), data.frag_block_offset);
return out;
}
/// Creates the symlink described by the inode.
/// Sets metadata.
fn extractSymlink(self: Inode, path: []const u8) !void {
const target = switch (self.data) {
.symlink => |s| s.target,
.ext_symlink => |s| s.target,
else => unreachable,
};
try std.fs.cwd().symLink(target, path, .{});
}
/// Creates the device described by the inode.
/// Sets metadata.
fn extractDeviceAndIPC(self: Inode, alloc: std.mem.Allocator, tables: *Tables, path: []const u8, options: ExtractionOptions) !void {
var mode: u32 = undefined;
var dev: u32 = 0;
switch (self.data) {
.char_dev => |d| {
mode = std.posix.S.IFCHR;
dev = d.dev;
},
.ext_char_dev => |d| {
mode = std.posix.S.IFCHR;
dev = d.dev;
},
.block_dev => |d| {
mode = std.posix.S.IFBLK;
dev = d.dev;
},
.ext_block_dev => |d| {
mode = std.posix.S.IFBLK;
dev = d.dev;
},
.fifo, .ext_fifo => mode = std.posix.S.IFIFO,
.socket, .ext_socket => mode = std.posix.S.IFSOCK,
else => unreachable,
}
const res: std.os.linux.E = @enumFromInt(std.os.linux.mknod(@ptrCast(path), mode, dev));
switch (res) {
.SUCCESS => {},
.ACCES => return std.fs.Dir.MakeError.AccessDenied,
.DQUOT => return std.fs.Dir.MakeError.DiskQuota,
.EXIST => return std.fs.Dir.MakeError.PathAlreadyExists,
.FAULT, .NOENT => return std.fs.Dir.MakeError.BadPathName,
.LOOP => return std.fs.Dir.MakeError.SymLinkLoop,
.NAMETOOLONG => return std.fs.Dir.MakeError.NameTooLong,
.NOMEM => return std.fs.Dir.MakeError.SystemResources,
.NOSPC => return std.fs.Dir.MakeError.NoSpaceLeft,
.NOTDIR => return std.fs.Dir.MakeError.NotDir,
.PERM => return std.fs.Dir.MakeError.PermissionDenied,
.ROFS => return std.fs.Dir.MakeError.ReadOnlyFileSystem,
else => return blk: {
std.debug.print("unhandled mknod result: {}\n", .{res});
break :blk std.fs.Dir.MakeError.Unexpected;
},
}
var fil = try std.fs.cwd().openFile(path, .{});
defer fil.close();
try self.setMetadata(alloc, tables, fil, options);
}
+101
View File
@@ -0,0 +1,101 @@
const std = @import("std");
const WaitGroup = std.Thread.WaitGroup;
const Mutex = std.Thread.Mutex;
const Archive = @import("../archive.zig");
const Inode = @import("../inode.zig");
const ExtractionOptions = @import("../options.zig");
const Tables = @import("../tables.zig");
const InodeFinish = @This();
const FinishEnum = enum {
wg,
fin,
};
pub const FinishUnion = union(FinishEnum) {
wg: *WaitGroup,
fin: *InodeFinish,
pub fn finish(self: FinishUnion) void {
switch (self) {
.wg => |wg| wg.finish(),
.fin => |fin| fin.finish(),
}
}
};
alloc: std.mem.Allocator,
inode: Inode,
path: []const u8,
tables: *Tables,
options: ExtractionOptions,
parent_finish: FinishUnion,
fil: ?std.fs.File,
out_err: *?anyerror,
wg: WaitGroup = .{},
mut: Mutex = .{},
pub fn create(
alloc: std.mem.Allocator,
inode: Inode,
path: []const u8,
tables: *Tables,
options: ExtractionOptions,
parent_finish: FinishUnion,
out_err: *?anyerror,
fil: ?std.fs.File,
work_size: usize,
) !*InodeFinish {
if (work_size == 0)
return error.InvalidWorkSize;
const out = try alloc.create(InodeFinish);
errdefer alloc.destroy(out);
out.* = .{
.alloc = alloc,
.inode = inode,
.path = path,
.tables = tables,
.options = options,
.parent_finish = parent_finish,
.out_err = out_err,
.fil = fil,
};
out.wg.startMany(work_size);
return out;
}
pub fn logError(self: *InodeFinish, comptime fmt: []const u8, args: anytype) void {
if (self.options.verbose)
self.options.verbose_writer.?.print(fmt, args) catch {};
}
pub fn finish(self: *InodeFinish) void {
self.mut.lock();
{
defer self.mut.unlock();
self.wg.finish();
if (!self.wg.isDone()) return;
}
defer {
self.parent_finish.finish();
self.alloc.destroy(self);
}
if (self.fil == null)
self.fil = std.fs.cwd().openFile(self.path, .{}) catch |err| {
if (self.options.verbose)
self.options.verbose_writer.?.print("Error opening {s} to set metadata: {}\n", .{ self.path, err }) catch {};
self.out_err.* = err;
return;
};
defer self.fil.?.close();
self.inode.setMetadata(self.alloc, self.tables, self.fil.?, self.options) catch |err| {
if (self.options.verbose)
self.options.verbose_writer.?.print("Error setting metadata to {s}: {}\n", .{ self.path, err }) catch {};
self.out_err.* = err;
return;
};
}
+41
View File
@@ -0,0 +1,41 @@
const std = @import("std");
const OffsetFile = @import("offset_file.zig");
const MetadataCache = @This();
alloc: std.mem.Allocator,
buf: []u8,
fixed_alloc: std.heap.FixedBufferAllocator,
cache: std.AutoArrayHashMap(u64, [8192]u8),
mut: std.Thread.Mutex = .{},
cache_mut: std.AutoArrayHashMap(u64, std.Thread.Mutex),
fil: OffsetFile,
pub fn init(alloc: std.mem.Allocator, cache_size: u64) !MetadataCache {}
pub fn deinit(self: *MetadataCache) void {
self.mut.lock();
defer self.mut.unlock();
self.cache.deinit();
self.cache_mut.deinit();
self.alloc.free(self.buf);
}
pub fn getChunk(self: *MetadataCache, offset: u64) ![8192]u8 {
var res = self.cache.get(offset);
if (res != null) return res.?;
var offset_mut = blk: {
self.mut.lock();
defer self.mut.unlock();
const mut = try self.cache_mut.getOrPut(offset);
if (!mut.found_existing)
mut.value_ptr.* = .{};
break :blk mut.value_ptr;
};
offset_mut.lock();
defer offset_mut.unlock();
}
+97
View File
@@ -0,0 +1,97 @@
const std = @import("std");
const Reader = std.Io.Reader;
const Writer = std.Io.Writer;
const Limit = std.Io.Limit;
const StreamError = std.Io.Reader.StreamError;
const DecompFn = @import("../decomp.zig").DecompFn;
const BlockHeader = packed struct {
size: u15,
uncompressed: bool,
};
const This = @This();
alloc: std.mem.Allocator,
rdr: *Reader,
decomp: DecompFn,
buf: [8192]u8 = undefined,
interface: Reader,
err: ?anyerror = null,
pub fn init(alloc: std.mem.Allocator, rdr: *Reader, decomp: DecompFn) This {
return .{
.alloc = alloc,
.rdr = rdr,
.decomp = decomp,
.interface = .{
.buffer = &[0]u8{},
.end = 0,
.seek = 0,
.vtable = &.{
.stream = stream,
.discard = discard,
.readVec = readVec,
},
},
};
}
fn advance(self: *This) !void {
self.interface.seek = 0;
var hdr: BlockHeader = undefined;
try self.rdr.readSliceEndian(BlockHeader, @ptrCast(&hdr), .little);
if (hdr.uncompressed) {
try self.rdr.readSliceEndian(u8, self.buf[0..hdr.size], .little);
self.interface.end = hdr.size;
self.interface.buffer = self.buf[0..hdr.size];
return;
}
var tmp_buf: [8192]u8 = undefined;
try self.rdr.readSliceAll(tmp_buf[0..hdr.size]);
self.interface.end = try self.decomp(self.alloc, tmp_buf[0..hdr.size], &self.buf);
self.interface.buffer = self.buf[0..self.interface.end];
}
fn stream(rdr: *Reader, wrt: *Writer, limit: Limit) StreamError!usize {
const self: *This = @fieldParentPtr("interface", rdr);
if (rdr.end == rdr.seek) self.advance() catch |err| {
self.err = err;
return StreamError.ReadFailed;
};
if (@intFromEnum(limit) == 0) return 0;
const to_write = @min(rdr.end - rdr.seek, @intFromEnum(limit));
const wrote = try wrt.write(self.buf[rdr.seek .. rdr.seek + to_write]);
self.interface.seek += wrote;
return wrote;
}
fn discard(rdr: *Reader, limit: Limit) error{ EndOfStream, ReadFailed }!usize {
const self: *This = @fieldParentPtr("interface", rdr);
if (rdr.end == rdr.seek) self.advance() catch |err| {
self.err = err;
return StreamError.ReadFailed;
};
if (@intFromEnum(limit) == 0) return 0;
const to_skip = @min(rdr.end - rdr.seek, @intFromEnum(limit));
rdr.seek += to_skip;
return to_skip;
}
fn readVec(rdr: *Reader, vec: [][]u8) error{ EndOfStream, ReadFailed }!usize {
const self: *This = @fieldParentPtr("interface", rdr);
if (rdr.end == rdr.seek) self.advance() catch |err| {
self.err = err;
return StreamError.ReadFailed;
};
var cur_red: usize = 0;
for (vec) |s| {
const to_copy: usize = @min(rdr.end - rdr.seek, s.len);
@memcpy(s[0..to_copy], self.buf[rdr.seek .. rdr.seek + to_copy]);
rdr.seek += to_copy;
cur_red += to_copy;
if (rdr.end == rdr.seek) break;
}
return cur_red;
}
+20
View File
@@ -0,0 +1,20 @@
//! A File where it's meaningful (to us) content starts at a given offset.
const std = @import("std");
const File = std.fs.File;
const Reader = std.fs.File.Reader;
const OffsetFile = @This();
fil: File,
offset: u64,
pub fn init(fil: File, init_offset: u64) OffsetFile {
return .{ .fil = fil, .offset = init_offset };
}
pub fn readerAt(self: OffsetFile, offset: u64, buffer: []u8) !Reader {
var rdr = self.fil.reader(buffer);
try rdr.seekTo(self.offset + offset);
return rdr;
}
+117
View File
@@ -0,0 +1,117 @@
const std = @import("std");
const DecompFn = @import("decomp.zig").DecompFn;
const Table = @import("tables.zig").Table;
const MetadataReader = @import("util/metadata.zig");
const OffsetFile = @import("util/offset_file.zig");
const Ref = packed struct {
block_offset: u16,
block_start: u32,
_: u16,
};
const Entry = packed struct {
ref: Ref,
count: u32,
size: u32,
};
const KeyPrefix = enum(u8) {
user,
trusted,
security,
};
const KeyRaw = packed struct {
type: packed struct {
prefix: KeyPrefix,
out_of_line: bool,
_: u7,
},
name_size: u16,
};
pub const KeyValue = struct {
key: [:0]u8,
value: []u8,
};
const XattrTable = @This();
alloc: std.mem.Allocator,
fil: OffsetFile,
decomp: DecompFn,
count: u32,
start: u64,
table: Table(Entry),
pub fn init(alloc: std.mem.Allocator, fil: OffsetFile, decomp: DecompFn, table_start: u64) !XattrTable {
var info = packed struct {
start: u64 = undefined,
count: u32 = undefined,
_: u32 = undefined,
}{};
var rdr = try fil.readerAt(table_start, &[0]u8{});
try rdr.interface.readSliceEndian(@TypeOf(info), @ptrCast(&info), .little);
return .{
.alloc = alloc,
.fil = fil,
.decomp = decomp,
.count = info.count,
.start = info.start,
.table = try .init(alloc, fil, decomp, table_start + 16, info.count),
};
}
pub fn deinit(self: *XattrTable) void {
self.table.deinit();
}
pub fn get(self: *XattrTable, alloc: std.mem.Allocator, idx: u32) ![]KeyValue {
const entry: Entry = try self.table.get(idx);
const out = try alloc.alloc(KeyValue, entry.count);
for (out) |*kv| {
var rdr = try self.fil.readerAt(self.start + entry.ref.block_start, &[0]u8{});
var meta: MetadataReader = .init(alloc, &rdr.interface, self.decomp);
try meta.interface.discardAll(entry.ref.block_offset);
var key_raw: KeyRaw = undefined;
try meta.interface.readSliceEndian(KeyRaw, @ptrCast(&key_raw), .little);
switch (key_raw.type.prefix) {
.user => {
kv.key = @ptrCast(try alloc.alloc(u8, key_raw.name_size + 5 + 1));
@memcpy(kv.key[0..5], "user.");
try meta.interface.readSliceAll(kv.key[5 .. kv.key.len - 1]);
kv.key[kv.key.len - 1] = 0;
},
.security => {
kv.key = @ptrCast(try alloc.alloc(u8, key_raw.name_size + 9 + 1));
@memcpy(kv.key[0..9], "security.");
try meta.interface.readSliceAll(kv.key[9 .. kv.key.len - 1]);
kv.key[kv.key.len - 1] = 0;
},
.trusted => {
kv.key = @ptrCast(try alloc.alloc(u8, key_raw.name_size + 8 + 1));
@memcpy(kv.key[0..8], "trusted.");
try meta.interface.readSliceAll(kv.key[8 .. kv.key.len - 1]);
kv.key[kv.key.len - 1] = 0;
},
}
if (key_raw.type.out_of_line) {
try meta.interface.discardAll(4);
var ref: Ref = undefined;
try meta.interface.readSliceEndian(Ref, @ptrCast(&ref), .little);
rdr = try self.fil.readerAt(self.start + ref.block_start, &[0]u8{});
meta = .init(alloc, &rdr.interface, self.decomp);
try meta.interface.discardAll(ref.block_offset);
}
var value_size: u32 = undefined;
try meta.interface.readSliceEndian(u32, @ptrCast(&value_size), .little);
kv.value = try alloc.alloc(u8, value_size);
try meta.interface.readSliceAll(kv.value);
}
return out;
}