Compare commits

..

30 Commits

Author SHA1 Message Date
Caleb Gardner f242de2710 Better disabling of compression types 2025-03-17 06:53:29 -05:00
Caleb Gardner 88315ee384 Fix build flags 2025-03-17 06:28:31 -05:00
Caleb Gardner 1e2a8f4b75 go mod tidy 2025-03-17 06:22:24 -05:00
Caleb Gardner 863b03fb19 Updated README 2025-03-17 06:21:19 -05:00
Caleb Gardner d3f84344d1 Fix build flags in lzma.go & xz.go 2025-03-17 06:19:45 -05:00
Caleb Gardner ad24995b7b Change no_lzma and no_lzo to no_obsolete and no_gpl
Added build tags section to README
2025-03-17 06:16:25 -05:00
Caleb Gardner 638355ab71 Merge pull request #37 from afbjorklund/comp-none
Allow disabling lzo and lzma
2025-03-17 05:45:26 -05:00
Anders F Björklund 04d914d403 Allow disabling lzo and lzma
By setting the buildtags "no_lzo" and/or "no_lzma",
one can drop the library dependency on lzo and lzma.

The same could be done for xz as well, but there are
still lots of archives using xz compression out there.
2025-03-16 13:56:04 +01:00
Caleb Gardner 7323fe56f6 Merge pull request #35 from afbjorklund/list
Add list option to unsquashfs
2025-03-15 15:48:41 -05:00
Caleb Gardner 6286da31e1 Merge branch 'main' into list 2025-03-15 15:48:06 -05:00
Caleb Gardner 77c87a9653 Merge pull request #34 from afbjorklund/unsquashfs-offset
Allow mounting with an offset
2025-03-15 15:34:04 -05:00
Anders F Björklund e6b0b83dcb Add support for uid/gid 2025-03-15 17:50:02 +01:00
Anders F Björklund cef9090210 Add support for symlinks 2025-03-15 17:46:02 +01:00
Anders F Björklund 24a9457c6b Refactor: export FileInfo 2025-03-15 17:35:40 +01:00
Anders F Björklund e0c1309ed4 Add list option to unsquashfs 2025-03-15 17:32:27 +01:00
Anders F Björklund 8b475b6cc4 Allow mounting with an offset
When mounting squashfs images embedded in apptainer image,
using offset means we don't need to use a temporary copy.
2025-03-15 17:26:59 +01:00
Caleb Gardner 3a48a0bcdc Remove t.Fatal at end of single file test 2025-03-12 00:11:29 -05:00
Caleb Gardner f11416493e Apply FileMode fixes to Inode.Mode() 2025-03-12 00:09:02 -05:00
Caleb Gardner 619bb023b1 Fix missing fileInfo.Mode() types 2025-03-12 00:03:58 -05:00
Caleb Gardner 38e4761d21 Merge pull request #33 from afbjorklund/fileinfo-symlink
Properly show symlinks in Mode
2025-03-11 23:51:01 -05:00
Anders F Björklund 06d2ef3056 Properly show symlinks in Mode
Previously they were extracted OK (as symlinks), but shown
as regular files with length 0 when getting the file info.
2025-03-11 18:46:31 +01:00
Caleb Gardner 446f29df70 Empty io.Reader buffer on EOF 2025-03-04 04:33:47 -06:00
Caleb Gardner d6c8efcfe6 Removed writeToWriteAt
Didn't seem to have any performance advantage
2025-03-04 04:08:13 -06:00
Caleb Gardner d890932d5c Use WriterAt if it's available for FullReader 2025-02-27 07:19:04 -06:00
Caleb Gardner 87b5ac7f5d gopls modernize 2025-02-27 02:46:22 -06:00
Caleb Gardner e9fdd89c67 Merge pull request #31 from willmurphyscode/main
fix: remove stray println
2024-12-10 16:09:45 -06:00
Will Murphy c80d150fdc fix: remove stray println 2024-12-10 17:00:57 -05:00
Caleb Gardner 03266d0560 Fix frag, id, inode table values on block boundries
Fixes bug mention in #30
2024-11-26 17:09:39 -06:00
Caleb Gardner 0f8a4e0027 Re-added NewReaderAtOffset 2024-09-20 20:10:33 -05:00
Caleb Gardner 2a33cad709 PERFORMANCE
Changed some struct values from pointers to normal values for improved performance.
2024-07-17 09:30:16 -05:00
23 changed files with 447 additions and 171 deletions
+8 -3
View File
@@ -11,6 +11,10 @@ Currently has support for reading squashfs files and extracting files and folder
Special thanks to <https://dr-emann.github.io/squashfs/> for some VERY important information in an easy to understand format. Special thanks to <https://dr-emann.github.io/squashfs/> for some VERY important information in an easy to understand format.
Thanks also to [distri's squashfs library](https://github.com/distr1/distri/tree/master/internal/squashfs) as I referenced it to figure some things out (and double check others). Thanks also to [distri's squashfs library](https://github.com/distr1/distri/tree/master/internal/squashfs) as I referenced it to figure some things out (and double check others).
## Build tags
As of `v1.1.0` this library has two optional build tags: `no_gpl` and `no_obsolete`. `no_gpl` disables the ability to read archives with lzo compression due to the library's gpl license. `no_obsolete` removes "obsolete" compression types for a reduced compilation size; currently this only disable lzma compression since it's superseded by xz.
## FUSE ## FUSE
As of `v1.0`, FUSE capabilities has been moved to [a separate library](https://github.com/CalebQ42/squashfuse). As of `v1.0`, FUSE capabilities has been moved to [a separate library](https://github.com/CalebQ42/squashfuse).
@@ -24,11 +28,12 @@ As of `v1.0`, FUSE capabilities has been moved to [a separate library](https://g
## Issues ## Issues
* Significantly slower then `unsquashfs` when extracting folders * Significantly slower then `unsquashfs` when nested images
* This seems to be related to above along with the general optimization of `unsquashfs` and it's compression libraries. * This seems to be related to above along with the general optimization of `unsquashfs` and it's compression libraries.
* Not to mention it's written in C
* Times seem to be largely dependent on file tree size and compression type. * Times seem to be largely dependent on file tree size and compression type.
* My main testing image (~100MB) using Zstd takes about 6x longer. * My main testing image (~100MB) using Zstd takes about 5x longer.
* An Arch Linux airootfs image (~780MB) using XZ compression with LZMA filters takes about 32x longer. * An Arch Linux airootfs image (~780MB) using XZ compression with LZMA filters takes about 30x longer.
* A Tensorflow docker image (~3.3GB) using Zstd takes about 12x longer. * A Tensorflow docker image (~3.3GB) using Zstd takes about 12x longer.
Note: These numbers are using `FastOptions()`. `DefaultOptions()` takes about 2x longer. Note: These numbers are using `FastOptions()`. `DefaultOptions()` takes about 2x longer.
+64 -1
View File
@@ -3,14 +3,62 @@ package main
import ( import (
"flag" "flag"
"fmt" "fmt"
"io/fs"
"os" "os"
"os/user"
"path/filepath"
"strconv"
"strings"
"time" "time"
"github.com/CalebQ42/squashfs" "github.com/CalebQ42/squashfs"
) )
func userName(uid int, numeric bool) string {
us := strconv.Itoa(uid)
if numeric {
return us
}
if u, err := user.LookupId(us); err == nil {
return u.Username
}
return us
}
func groupName(gid int, numeric bool) string {
gs := strconv.Itoa(gid)
if numeric {
return gs
}
if g, err := user.LookupGroupId(gs); err == nil {
return g.Name
}
return gs
}
func printEntry(root, path string, d fs.DirEntry, numeric bool) {
fi, _ := d.Info()
sfi := fi.(squashfs.FileInfo)
owner := fmt.Sprintf("%s/%s",
userName(sfi.Uid(), numeric),
groupName(sfi.Gid(), numeric))
link := ""
if sfi.IsSymlink() {
link = " -> " + sfi.SymlinkPath()
}
fmt.Printf("%s %s %*d %s %s%s\n",
strings.ToLower(fi.Mode().String()),
owner, 26-len(owner), fi.Size(),
fi.ModTime().Format("2006-01-02 15:04"),
filepath.Join(root, path), link)
}
func main() { func main() {
verbose := flag.Bool("v", false, "Verbose") verbose := flag.Bool("v", false, "Verbose")
list := flag.Bool("l", false, "List")
long := flag.Bool("ll", false, "List with attributes")
numeric := flag.Bool("lln", false, "List with attributes and numeric ids")
offset := flag.Int64("o", 0, "Offset")
ignore := flag.Bool("ip", false, "Ignore Permissions and extract all files/folders with 0755") ignore := flag.Bool("ip", false, "Ignore Permissions and extract all files/folders with 0755")
flag.Parse() flag.Parse()
if len(flag.Args()) < 2 { if len(flag.Args()) < 2 {
@@ -21,10 +69,25 @@ func main() {
if err != nil { if err != nil {
panic(err) panic(err)
} }
r, err := squashfs.NewReader(f) r, err := squashfs.NewReaderAtOffset(f, *offset)
if err != nil { if err != nil {
panic(err) panic(err)
} }
if *list || *long || *numeric {
root := flag.Arg(1)
fs.WalkDir(r, ".", func(path string, d fs.DirEntry, err error) error {
if err != nil {
panic(err)
}
if *long || *numeric {
printEntry(root, path, d, *numeric)
} else {
fmt.Println(filepath.Join(root, path))
}
return nil
})
return
}
op := squashfs.DefaultOptions() op := squashfs.DefaultOptions()
op.Verbose = *verbose op.Verbose = *verbose
op.IgnorePerm = *ignore op.IgnorePerm = *ignore
+28 -18
View File
@@ -19,16 +19,16 @@ import (
// File represents a file inside a squashfs archive. // File represents a file inside a squashfs archive.
type File struct { type File struct {
b *squashfslow.FileBase
full *data.FullReader full *data.FullReader
rdr *data.Reader rdr *data.Reader
parent *FS parent *FS
r *Reader r *Reader
b squashfslow.FileBase
dirsRead int dirsRead int
} }
// Creates a new *File from the given *squashfs.Base // Creates a new *File from the given *squashfs.Base
func (r *Reader) FileFromBase(b *squashfslow.FileBase, parent *FS) *File { func (r *Reader) FileFromBase(b squashfslow.FileBase, parent *FS) *File {
return &File{ return &File{
b: b, b: b,
parent: parent, parent: parent,
@@ -40,7 +40,7 @@ func (f *File) FS() (*FS, error) {
if !f.IsDir() { if !f.IsDir() {
return nil, errors.New("not a directory") return nil, errors.New("not a directory")
} }
d, err := f.b.ToDir(f.r.Low) d, err := f.b.ToDir(&f.r.Low)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -114,7 +114,7 @@ func (f *File) ReadDir(n int) ([]fs.DirEntry, error) {
if !f.IsDir() { if !f.IsDir() {
return nil, errors.New("file is not a directory") return nil, errors.New("file is not a directory")
} }
d, err := f.b.ToDir(f.r.Low) d, err := f.b.ToDir(&f.r.Low)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -127,7 +127,7 @@ func (f *File) ReadDir(n int) ([]fs.DirEntry, error) {
} }
} }
var out []fs.DirEntry var out []fs.DirEntry
var fi fileInfo var fi FileInfo
for _, e := range d.Entries[start:end] { for _, e := range d.Entries[start:end] {
fi, err = f.r.newFileInfo(e) fi, err = f.r.newFileInfo(e)
if err != nil { if err != nil {
@@ -142,7 +142,15 @@ func (f *File) ReadDir(n int) ([]fs.DirEntry, error) {
// Returns the file's fs.FileInfo // Returns the file's fs.FileInfo
func (f *File) Stat() (fs.FileInfo, error) { func (f *File) Stat() (fs.FileInfo, error) {
return newFileInfo(f.b.Name, f.b.Inode), nil uid, err := f.b.Uid(&f.r.Low)
if err != nil {
return nil, err
}
gid, err := f.b.Gid(&f.r.Low)
if err != nil {
return nil, err
}
return newFileInfo(f.b.Name, uid, gid, &f.b.Inode), nil
} }
// SymlinkPath returns the symlink's target path. Is the File isn't a symlink, returns an empty string. // SymlinkPath returns the symlink's target path. Is the File isn't a symlink, returns an empty string.
@@ -173,15 +181,16 @@ func (f *File) WriteTo(w io.Writer) (int64, error) {
func (f *File) initializeReaders() error { func (f *File) initializeReaders() error {
var err error var err error
f.rdr, f.full, err = f.b.GetRegFileReaders(f.r.Low) f.rdr, f.full, err = f.b.GetRegFileReaders(&f.r.Low)
return err return err
} }
func (f *File) deviceDevices() (maj uint32, min uint32) { func (f *File) deviceDevices() (maj uint32, min uint32) {
var dev uint32 var dev uint32
if f.b.Inode.Type == inode.Char || f.b.Inode.Type == inode.Block { switch f.b.Inode.Type {
case inode.Char, inode.Block:
dev = f.b.Inode.Data.(inode.Device).Dev dev = f.b.Inode.Data.(inode.Device).Dev
} else if f.b.Inode.Type == inode.EChar || f.b.Inode.Type == inode.EBlock { case inode.EChar, inode.EBlock:
dev = f.b.Inode.Data.(inode.EDevice).Dev dev = f.b.Inode.Data.(inode.EDevice).Dev
} }
return dev >> 8, dev & 0x000FF return dev >> 8, dev & 0x000FF
@@ -218,7 +227,7 @@ func (f *File) ExtractWithOptions(path string, op *ExtractionOptions) error {
} }
switch f.b.Inode.Type { switch f.b.Inode.Type {
case inode.Dir, inode.EDir: case inode.Dir, inode.EDir:
d, err := f.b.ToDir(f.r.Low) d, err := f.b.ToDir(&f.r.Low)
if err != nil { if err != nil {
if op.Verbose { if op.Verbose {
log.Println("Failed to create squashfs.Directory for", path) log.Println("Failed to create squashfs.Directory for", path)
@@ -234,7 +243,7 @@ func (f *File) ExtractWithOptions(path string, op *ExtractionOptions) error {
} }
return errors.Join(errors.New("failed to get base from entry: "+path), err) return errors.Join(errors.New("failed to get base from entry: "+path), err)
} }
go func(b *squashfslow.FileBase, path string) { go func(b squashfslow.FileBase, path string) {
i := op.manager.Lock() i := op.manager.Lock()
if b.IsDir() { if b.IsDir() {
extDir := filepath.Join(path, b.Name) extDir := filepath.Join(path, b.Name)
@@ -266,7 +275,7 @@ func (f *File) ExtractWithOptions(path string, op *ExtractionOptions) error {
}(b, path) }(b, path)
} }
var errCache []error var errCache []error
for i := 0; i < len(d.Entries); i++ { for range d.Entries {
err := <-errChan err := <-errChan
if err != nil { if err != nil {
errCache = append(errCache, err) errCache = append(errCache, err)
@@ -285,7 +294,7 @@ func (f *File) ExtractWithOptions(path string, op *ExtractionOptions) error {
return errors.Join(errors.New("failed to create file: "+path), err) return errors.Join(errors.New("failed to create file: "+path), err)
} }
defer outFil.Close() defer outFil.Close()
full, err := f.b.GetFullReader(f.r.Low) full, err := f.b.GetFullReader(&f.r.Low)
if err != nil { if err != nil {
if op.Verbose { if op.Verbose {
log.Println("Failed to create full reader for", path) log.Println("Failed to create full reader for", path)
@@ -363,11 +372,12 @@ func (f *File) ExtractWithOptions(path string, op *ExtractionOptions) error {
} }
path = filepath.Join(path, f.b.Name) path = filepath.Join(path, f.b.Name)
var typ string var typ string
if f.b.Inode.Type == inode.Char || f.b.Inode.Type == inode.EChar { switch f.b.Inode.Type {
case inode.Char, inode.EChar:
typ = "c" typ = "c"
} else if f.b.Inode.Type == inode.Block || f.b.Inode.Type == inode.EBlock { case inode.Block, inode.EBlock:
typ = "b" typ = "b"
} else { //Fifo IPC default: //Fifo IPC
if runtime.GOOS == "darwin" { if runtime.GOOS == "darwin" {
if op.Verbose { if op.Verbose {
log.Println(f.path(), "ignored. A Fifo file and can't be created on Darwin.") log.Println(f.path(), "ignored. A Fifo file and can't be created on Darwin.")
@@ -406,7 +416,7 @@ func (f *File) ExtractWithOptions(path string, op *ExtractionOptions) error {
if op.IgnorePerm { if op.IgnorePerm {
return nil return nil
} }
uid, err := f.b.Uid(f.r.Low) uid, err := f.b.Uid(&f.r.Low)
if err != nil { if err != nil {
if op.Verbose { if op.Verbose {
log.Println("Failed to get uid for", path) log.Println("Failed to get uid for", path)
@@ -414,7 +424,7 @@ func (f *File) ExtractWithOptions(path string, op *ExtractionOptions) error {
} }
return nil return nil
} }
gid, err := f.b.Gid(f.r.Low) gid, err := f.b.Gid(&f.r.Low)
if err != nil { if err != nil {
if op.Verbose { if op.Verbose {
log.Println("Failed to get gid for", path) log.Println("Failed to get gid for", path)
+74 -16
View File
@@ -8,61 +8,119 @@ import (
"github.com/CalebQ42/squashfs/low/inode" "github.com/CalebQ42/squashfs/low/inode"
) )
type fileInfo struct { type FileInfo struct {
name string name string
uid uint32
gid uint32
size int64 size int64
target string
perm uint32 perm uint32
modTime uint32 modTime uint32
fileType uint16 fileType uint16
} }
func (r Reader) newFileInfo(e directory.Entry) (fileInfo, error) { func (r Reader) newFileInfo(e directory.Entry) (FileInfo, error) {
i, err := r.Low.InodeFromEntry(e) b, err := r.Low.BaseFromEntry(e)
if err != nil { if err != nil {
return fileInfo{}, err return FileInfo{}, err
} }
return newFileInfo(e.Name, i), nil uid, err := b.Uid(&r.Low)
if err != nil {
return FileInfo{}, err
}
gid, err := b.Gid(&r.Low)
if err != nil {
return FileInfo{}, err
}
return newFileInfo(e.Name, uid, gid, &b.Inode), nil
} }
func newFileInfo(name string, i *inode.Inode) fileInfo { func newFileInfo(name string, uid, gid uint32, i *inode.Inode) FileInfo {
var size int64 var size int64
if i.Type == inode.Fil { var target string
switch i.Type {
case inode.Fil:
size = int64(i.Data.(inode.File).Size) size = int64(i.Data.(inode.File).Size)
} else if i.Type == inode.EFil { case inode.EFil:
size = int64(i.Data.(inode.EFile).Size) size = int64(i.Data.(inode.EFile).Size)
case inode.Sym:
target = string(i.Data.(inode.Symlink).Target)
case inode.ESym:
target = string(i.Data.(inode.ESymlink).Target)
} }
return fileInfo{ return FileInfo{
name: name, name: name,
uid: uid,
gid: gid,
size: size, size: size,
target: target,
perm: uint32(i.Perm), perm: uint32(i.Perm),
modTime: i.ModTime, modTime: i.ModTime,
fileType: i.Type, fileType: i.Type,
} }
} }
func (f fileInfo) Name() string { func (f FileInfo) Name() string {
return f.name return f.name
} }
func (f fileInfo) Size() int64 { func (f FileInfo) Uid() int {
return int(f.uid)
}
func (f FileInfo) Gid() int {
return int(f.gid)
}
func (f FileInfo) Size() int64 {
return f.size return f.size
} }
func (f fileInfo) Mode() fs.FileMode { func (f FileInfo) SymlinkPath() string {
if f.IsDir() { return f.target
}
func (f FileInfo) Mode() fs.FileMode {
switch f.fileType {
case inode.Dir, inode.EDir:
return fs.FileMode(f.perm | uint32(fs.ModeDir)) return fs.FileMode(f.perm | uint32(fs.ModeDir))
case inode.Sym, inode.ESym:
return fs.FileMode(f.perm | uint32(fs.ModeSymlink))
case inode.Char, inode.EChar, inode.Block, inode.EBlock:
return fs.FileMode(f.perm | uint32(fs.ModeDevice))
case inode.Fifo, inode.EFifo:
return fs.FileMode(f.perm | uint32(fs.ModeNamedPipe))
case inode.Sock, inode.ESock:
return fs.FileMode(f.perm | uint32(fs.ModeSocket))
} }
return fs.FileMode(f.perm) return fs.FileMode(f.perm)
} }
func (f fileInfo) ModTime() time.Time { func (f FileInfo) ModTime() time.Time {
return time.Unix(int64(f.modTime), 0) return time.Unix(int64(f.modTime), 0)
} }
func (f fileInfo) IsDir() bool { func (f FileInfo) IsDir() bool {
return f.fileType == inode.Dir || f.fileType == inode.EDir return f.fileType == inode.Dir || f.fileType == inode.EDir
} }
func (f fileInfo) Sys() any { func (f FileInfo) IsSymlink() bool {
return f.fileType == inode.Sym || f.fileType == inode.ESym
}
func (f FileInfo) IsDevice() bool {
return f.fileType == inode.Block || f.fileType == inode.EBlock ||
f.fileType == inode.Char || f.fileType == inode.EChar
}
func (f FileInfo) IsFifo() bool {
return f.fileType == inode.Fifo || f.fileType == inode.EFifo
}
func (f FileInfo) IsSocket() bool {
return f.fileType == inode.Sock || f.fileType == inode.ESock
}
func (f FileInfo) Sys() any {
return nil return nil
} }
+6 -6
View File
@@ -15,13 +15,13 @@ import (
// FS is a fs.FS representation of a squashfs directory. // FS is a fs.FS representation of a squashfs directory.
// Implements fs.GlobFS, fs.ReadDirFS, fs.ReadFileFS, fs.StatFS, and fs.SubFS // Implements fs.GlobFS, fs.ReadDirFS, fs.ReadFileFS, fs.StatFS, and fs.SubFS
type FS struct { type FS struct {
d *squashfslow.Directory
r *Reader r *Reader
parent *FS parent *FS
d squashfslow.Directory
} }
// Creates a new *FS from the given squashfs.directory // Creates a new *FS from the given squashfs.directory
func (r *Reader) FSFromDirectory(d *squashfslow.Directory, parent *FS) *FS { func (r *Reader) FSFromDirectory(d squashfslow.Directory, parent *FS) *FS {
return &FS{ return &FS{
d: d, d: d,
r: r, r: r,
@@ -42,7 +42,7 @@ func (f *FS) Glob(pattern string) (out []string, err error) {
} }
} }
split := strings.Split(pattern, "/") split := strings.Split(pattern, "/")
for i := 0; i < len(f.d.Entries); i++ { for i := range f.d.Entries {
if match, _ := path.Match(split[0], f.d.Entries[i].Name); match { if match, _ := path.Match(split[0], f.d.Entries[i].Name); match {
if len(split) == 1 { if len(split) == 1 {
out = append(out, f.d.Entries[i].Name) out = append(out, f.d.Entries[i].Name)
@@ -80,7 +80,7 @@ func (f *FS) Glob(pattern string) (out []string, err error) {
Err: err, Err: err,
} }
} }
for i := 0; i < len(subGlob); i++ { for i := range subGlob {
subGlob[i] = f.d.Name + "/" + subGlob[i] subGlob[i] = f.d.Name + "/" + subGlob[i]
} }
out = append(out, subGlob...) out = append(out, subGlob...)
@@ -142,7 +142,7 @@ func (f *FS) Open(name string) (fs.File, error) {
Err: fs.ErrNotExist, Err: fs.ErrNotExist,
} }
} }
d, err := b.ToDir(f.r.Low) d, err := b.ToDir(&f.r.Low)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -255,7 +255,7 @@ func (f *FS) ExtractWithOptions(folder string, op *ExtractionOptions) error {
// Returns the FS as a *File // Returns the FS as a *File
func (f *FS) File() *File { func (f *FS) File() *File {
return &File{ return &File{
b: &f.d.FileBase, b: f.d.FileBase,
parent: f.parent, parent: f.parent,
r: f.r, r: f.r,
} }
+4 -4
View File
@@ -1,11 +1,11 @@
module github.com/CalebQ42/squashfs module github.com/CalebQ42/squashfs
go 1.21.5 go 1.24.0
require ( require (
github.com/pierrec/lz4/v4 v4.1.19 github.com/klauspost/compress v1.18.0
github.com/ulikunitz/xz v0.5.11 github.com/pierrec/lz4/v4 v4.1.22
github.com/klauspost/compress v1.17.4
github.com/rasky/go-lzo v0.0.0-20200203143853-96a758eda86e github.com/rasky/go-lzo v0.0.0-20200203143853-96a758eda86e
github.com/therootcompany/xz v1.0.1 github.com/therootcompany/xz v1.0.1
github.com/ulikunitz/xz v0.5.12
) )
+6 -6
View File
@@ -1,10 +1,10 @@
github.com/klauspost/compress v1.17.4 h1:Ej5ixsIri7BrIjBkRZLTo6ghwrEtHFk7ijlczPW4fZ4= github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.17.4/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM= github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/pierrec/lz4/v4 v4.1.19 h1:tYLzDnjDXh9qIxSTKHwXwOYmm9d887Y7Y1ZkyXYHAN4= github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.19/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4= github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/rasky/go-lzo v0.0.0-20200203143853-96a758eda86e h1:dCWirM5F3wMY+cmRda/B1BiPsFtmzXqV9b0hLWtVBMs= github.com/rasky/go-lzo v0.0.0-20200203143853-96a758eda86e h1:dCWirM5F3wMY+cmRda/B1BiPsFtmzXqV9b0hLWtVBMs=
github.com/rasky/go-lzo v0.0.0-20200203143853-96a758eda86e/go.mod h1:9leZcVcItj6m9/CfHY5Em/iBrCz7js8LcRQGTKEEv2M= github.com/rasky/go-lzo v0.0.0-20200203143853-96a758eda86e/go.mod h1:9leZcVcItj6m9/CfHY5Em/iBrCz7js8LcRQGTKEEv2M=
github.com/therootcompany/xz v1.0.1 h1:CmOtsn1CbtmyYiusbfmhmkpAAETj0wBIH6kCYaX+xzw= github.com/therootcompany/xz v1.0.1 h1:CmOtsn1CbtmyYiusbfmhmkpAAETj0wBIH6kCYaX+xzw=
github.com/therootcompany/xz v1.0.1/go.mod h1:3K3UH1yCKgBneZYhuQUvJ9HPD19UEXEI0BWbMn8qNMY= github.com/therootcompany/xz v1.0.1/go.mod h1:3K3UH1yCKgBneZYhuQUvJ9HPD19UEXEI0BWbMn8qNMY=
github.com/ulikunitz/xz v0.5.11 h1:kpFauv27b6ynzBNT/Xy+1k+fK4WswhN/6PN5WhFAGw8= github.com/ulikunitz/xz v0.5.12 h1:37Nm15o69RwBkXM0J6A5OlE67RZTfzUxTj8fB3dfcsc=
github.com/ulikunitz/xz v0.5.11/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14= github.com/ulikunitz/xz v0.5.12/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
+6
View File
@@ -1,3 +1,5 @@
//go:build !no_obsolete
package decompress package decompress
import ( import (
@@ -9,6 +11,10 @@ import (
type Lzma struct{} type Lzma struct{}
func NewLzma() (Lzma, error) {
return Lzma{}, nil
}
func (l Lzma) Decompress(data []byte) ([]byte, error) { func (l Lzma) Decompress(data []byte) ([]byte, error) {
rdr, err := lzma.NewReader(bytes.NewReader(data)) rdr, err := lzma.NewReader(bytes.NewReader(data))
if err != nil { if err != nil {
+17
View File
@@ -0,0 +1,17 @@
//go:build no_obsolete
package decompress
import (
"errors"
)
type Lzma struct{}
func NewLzma() (Lzma, error) {
return Lzma{}, errors.New("lzma compression is disable in this build with no_obsolete")
}
func (l Lzma) Decompress(data []byte) ([]byte, error) {
return nil, errors.New("lzma compression is disable in this build with no_obsolete")
}
+6
View File
@@ -1,3 +1,5 @@
//go:build !no_gpl
package decompress package decompress
import ( import (
@@ -8,6 +10,10 @@ import (
type Lzo struct{} type Lzo struct{}
func NewLzo() (Lzo, error) {
return Lzo{}, nil
}
func (l Lzo) Decompress(data []byte) ([]byte, error) { func (l Lzo) Decompress(data []byte) ([]byte, error) {
return lzo.Decompress1X(bytes.NewReader(data), len(data), 0) return lzo.Decompress1X(bytes.NewReader(data), len(data), 0)
} }
+15
View File
@@ -0,0 +1,15 @@
//go:build no_gpl
package decompress
import "errors"
type Lzo struct{}
func NewLzo() (Lzo, error) {
return Lzo{}, errors.New("lzo compression is disable in this build with no_gpl")
}
func (l Lzo) Decompress(data []byte) ([]byte, error) {
return nil, errors.New("lzo compression is disable in this build with no_gpl")
}
+1 -4
View File
@@ -50,10 +50,7 @@ func (r *Reader) Read(b []byte) (int, error) {
return curRead, err return curRead, err
} }
} }
toRead = len(b) - curRead toRead = min(len(b)-curRead, len(r.dat)-int(r.curOffset))
if toRead > len(r.dat)-int(r.curOffset) {
toRead = len(r.dat) - int(r.curOffset)
}
copy(b[curRead:], r.dat[r.curOffset:int(r.curOffset)+toRead]) copy(b[curRead:], r.dat[r.curOffset:int(r.curOffset)+toRead])
r.curOffset += uint16(toRead) r.curOffset += uint16(toRead)
curRead += toRead curRead += toRead
+19
View File
@@ -0,0 +1,19 @@
package toreader
import "io"
type OffsetReader struct {
r io.ReaderAt
off int64
}
func NewOffsetReader(r io.ReaderAt, off int64) *OffsetReader {
return &OffsetReader{
r: r,
off: off,
}
}
func (r OffsetReader) ReadAt(p []byte, off int64) (n int, e error) {
return r.r.ReadAt(p, off+r.off)
}
+90 -16
View File
@@ -18,7 +18,6 @@ type FullReader struct {
r io.ReaderAt r io.ReaderAt
d decompress.Decompressor d decompress.Decompressor
frag FragReaderConstructor frag FragReaderConstructor
retPool *sync.Pool
sizes []uint32 sizes []uint32
initialOffset int64 initialOffset int64
finalBlockSize uint64 finalBlockSize uint64
@@ -35,11 +34,6 @@ func NewFullReader(r io.ReaderAt, initialOffset int64, d decompress.Decompressor
goroutineLimit: uint16(runtime.NumCPU()), goroutineLimit: uint16(runtime.NumCPU()),
finalBlockSize: finalBlockSize, finalBlockSize: finalBlockSize,
blockSize: blockSize, blockSize: blockSize,
retPool: &sync.Pool{
New: func() any {
return &retValue{}
},
},
} }
} }
@@ -48,6 +42,9 @@ func (r *FullReader) AddFrag(frag FragReaderConstructor) {
} }
func (r *FullReader) SetGoroutineLimit(limit uint16) { func (r *FullReader) SetGoroutineLimit(limit uint16) {
if limit <= 0 {
r.goroutineLimit = 1
}
r.goroutineLimit = limit r.goroutineLimit = limit
} }
@@ -57,8 +54,8 @@ type retValue struct {
index uint64 index uint64
} }
func (r *FullReader) process(index uint64, fileOffset uint64, retChan chan *retValue) { func (r FullReader) process(index uint64, fileOffset uint64, pool *sync.Pool, retChan chan *retValue) {
ret := r.retPool.Get().(*retValue) ret := pool.Get().(*retValue)
ret.index = index ret.index = index
realSize := r.sizes[index] &^ (1 << 24) realSize := r.sizes[index] &^ (1 << 24)
if realSize == 0 { if realSize == 0 {
@@ -79,7 +76,10 @@ func (r *FullReader) process(index uint64, fileOffset uint64, retChan chan *retV
retChan <- ret retChan <- ret
} }
func (r *FullReader) WriteTo(w io.Writer) (int64, error) { func (r FullReader) WriteTo(w io.Writer) (int64, error) {
// if wa, is := w.(io.WriterAt); is {
// return r.writeToWriteAt(wa)
// }
var curIndex uint64 var curIndex uint64
var curOffset uint64 var curOffset uint64
var toProcess uint16 var toProcess uint16
@@ -87,14 +87,16 @@ func (r *FullReader) WriteTo(w io.Writer) (int64, error) {
cache := make(map[uint64]*retValue) cache := make(map[uint64]*retValue)
var errCache []error var errCache []error
retChan := make(chan *retValue, r.goroutineLimit) retChan := make(chan *retValue, r.goroutineLimit)
for i := uint64(0); i < uint64(math.Ceil(float64(len(r.sizes))/float64(r.goroutineLimit))); i++ { pool := &sync.Pool{
toProcess = uint16(len(r.sizes)) - (uint16(i) * r.goroutineLimit) New: func() any {
if toProcess > r.goroutineLimit { return &retValue{}
toProcess = r.goroutineLimit },
} }
for i := uint64(0); i < uint64(math.Ceil(float64(len(r.sizes))/float64(r.goroutineLimit))); i++ {
toProcess = min(uint16(len(r.sizes))-(uint16(i)*r.goroutineLimit), r.goroutineLimit)
// Start all the goroutines // Start all the goroutines
for j := uint16(0); j < toProcess; j++ { for j := uint16(0); j < toProcess; j++ {
go r.process((i*uint64(r.goroutineLimit))+uint64(j), curOffset, retChan) go r.process((i*uint64(r.goroutineLimit))+uint64(j), curOffset, pool, retChan)
curOffset += uint64(r.sizes[(i*uint64(r.goroutineLimit))+uint64(j)]) &^ (1 << 24) curOffset += uint64(r.sizes[(i*uint64(r.goroutineLimit))+uint64(j)]) &^ (1 << 24)
} }
// Then consume the results on retChan // Then consume the results on retChan
@@ -128,7 +130,7 @@ func (r *FullReader) WriteTo(w io.Writer) (int64, error) {
} }
continue continue
} }
r.retPool.Put(res) pool.Put(res)
curIndex++ curIndex++
// Now we recursively try to clear the cache // Now we recursively try to clear the cache
for len(cache) > 0 { for len(cache) > 0 {
@@ -146,7 +148,7 @@ func (r *FullReader) WriteTo(w io.Writer) (int64, error) {
break break
} }
delete(cache, curIndex) delete(cache, curIndex)
r.retPool.Put(res) pool.Put(res)
curIndex++ curIndex++
} }
} }
@@ -172,3 +174,75 @@ func (r *FullReader) WriteTo(w io.Writer) (int64, error) {
} }
return wrote, nil return wrote, nil
} }
// func (r FullReader) writeToWriteAt(w io.WriterAt) (out int64, outErr error) {
// wait := &sync.WaitGroup{}
// wait.Add(len(r.sizes))
// mgr := routinemanager.NewManager(r.goroutineLimit)
// curOffset := r.initialOffset
// for i := uint64(0); i < uint64(len(r.sizes)); i++ {
// go func(index uint64, fileOffset int64) {
// lckNum := mgr.Lock()
// defer mgr.Unlock(lckNum)
// defer wait.Done()
// realSize := r.sizes[index] &^ (1 << 24)
// if realSize == 0 {
// if index == uint64(len(r.sizes))-1 && r.frag == nil {
// _, err := w.WriteAt([]byte{0}, int64((uint64(r.blockSize)*index)+r.finalBlockSize)-1)
// if err != nil {
// outErr = errors.Join(outErr, err)
// return
// }
// out = max(out, int64((uint64(r.blockSize)*index)+r.finalBlockSize))
// }
// return
// }
// data := make([]byte, realSize)
// err := binary.Read(toreader.NewReader(r.r, int64(fileOffset)), binary.LittleEndian, &data)
// if err != nil {
// outErr = errors.Join(outErr, err)
// return
// }
// if r.sizes[index] == realSize {
// data, err = r.d.Decompress(data)
// }
// if err != nil {
// outErr = errors.Join(outErr, err)
// return
// }
// _, err = w.WriteAt(data, int64(uint64(r.blockSize)*index))
// if err != nil {
// outErr = errors.Join(outErr, err)
// return
// }
// out = max(out, int64(uint64(r.blockSize)*(index+1)))
// }(i, curOffset)
// curOffset += int64(r.sizes[i]) &^ (1 << 24)
// }
// if r.frag != nil {
// wait.Add(1)
// go func() {
// lckNum := mgr.Lock()
// defer mgr.Unlock(lckNum)
// defer wait.Done()
// rdr, err := r.frag()
// if err != nil {
// outErr = errors.Join(outErr, err)
// return
// }
// dat, err := io.ReadAll(rdr)
// if err != nil {
// outErr = errors.Join(outErr, err)
// return
// }
// _, err = w.WriteAt(dat, int64(int(r.blockSize)*len(r.sizes)))
// if err != nil {
// outErr = errors.Join(outErr, err)
// return
// }
// out = int64(int(r.blockSize)*len(r.sizes)) + int64(r.finalBlockSize)
// }()
// }
// wait.Wait()
// return
// }
+2 -4
View File
@@ -41,6 +41,7 @@ func (r *Reader) advance() error {
r.dat, err = io.ReadAll(r.frag) r.dat, err = io.ReadAll(r.frag)
return err return err
} else if r.curIndex >= uint64(len(r.sizes)) { } else if r.curIndex >= uint64(len(r.sizes)) {
r.dat = []byte{}
return io.EOF return io.EOF
} }
realSize := r.sizes[r.curIndex] &^ (1 << 24) realSize := r.sizes[r.curIndex] &^ (1 << 24)
@@ -73,10 +74,7 @@ func (r *Reader) Read(b []byte) (int, error) {
return curRead, err return curRead, err
} }
} }
toRead = len(b) - curRead toRead = min(len(b)-curRead, len(r.dat)-r.curOffset)
if toRead > len(r.dat)-r.curOffset {
toRead = len(r.dat) - r.curOffset
}
toRead = copy(b[curRead:], r.dat[r.curOffset:r.curOffset+toRead]) toRead = copy(b[curRead:], r.dat[r.curOffset:r.curOffset+toRead])
r.curOffset += toRead r.curOffset += toRead
curRead += toRead curRead += toRead
+13 -13
View File
@@ -18,10 +18,10 @@ type Directory struct {
Entries []directory.Entry Entries []directory.Entry
} }
func (r *Reader) directoryFromRef(ref uint64, name string) (*Directory, error) { func (r *Reader) directoryFromRef(ref uint64, name string) (Directory, error) {
i, err := r.InodeFromRef(ref) i, err := r.InodeFromRef(ref)
if err != nil { if err != nil {
return nil, err return Directory{}, err
} }
var blockStart uint32 var blockStart uint32
var size uint32 var size uint32
@@ -36,48 +36,48 @@ func (r *Reader) directoryFromRef(ref uint64, name string) (*Directory, error) {
size = i.Data.(inode.EDirectory).Size size = i.Data.(inode.EDirectory).Size
offset = i.Data.(inode.EDirectory).Offset offset = i.Data.(inode.EDirectory).Offset
default: default:
return nil, errors.New("not a directory") return Directory{}, errors.New("not a directory")
} }
dirRdr := metadata.NewReader(toreader.NewReader(r.r, int64(r.Superblock.DirTableStart)+int64(blockStart)), r.d) dirRdr := metadata.NewReader(toreader.NewReader(r.r, int64(r.Superblock.DirTableStart)+int64(blockStart)), r.d)
defer dirRdr.Close() defer dirRdr.Close()
_, err = dirRdr.Read(make([]byte, offset)) _, err = dirRdr.Read(make([]byte, offset))
if err != nil { if err != nil {
return nil, err return Directory{}, err
} }
entries, err := directory.ReadDirectory(dirRdr, size) entries, err := directory.ReadDirectory(dirRdr, size)
if err != nil { if err != nil {
return nil, err return Directory{}, err
} }
return &Directory{ return Directory{
FileBase: *r.BaseFromInode(i, name), FileBase: r.BaseFromInode(i, name),
Entries: entries, Entries: entries,
}, nil }, nil
} }
func (d *Directory) Open(r *Reader, path string) (*FileBase, error) { func (d *Directory) Open(r *Reader, path string) (FileBase, error) {
path = filepath.Clean(path) path = filepath.Clean(path)
if path == "." || path == "" { if path == "." || path == "" {
return &d.FileBase, nil return d.FileBase, nil
} }
split := strings.Split(path, "/") split := strings.Split(path, "/")
i, found := slices.BinarySearchFunc(d.Entries, split[0], func(e directory.Entry, name string) int { i, found := slices.BinarySearchFunc(d.Entries, split[0], func(e directory.Entry, name string) int {
return strings.Compare(e.Name, name) return strings.Compare(e.Name, name)
}) })
if !found { if !found {
return nil, fs.ErrNotExist return FileBase{}, fs.ErrNotExist
} }
b, err := r.BaseFromEntry(d.Entries[i]) b, err := r.BaseFromEntry(d.Entries[i])
if err != nil { if err != nil {
return nil, err return FileBase{}, err
} }
if len(split) == 1 { if len(split) == 1 {
return b, nil return b, nil
} else if !b.IsDir() { } else if !b.IsDir() {
return nil, fs.ErrNotExist return FileBase{}, fs.ErrNotExist
} }
dir, err := b.ToDir(r) dir, err := b.ToDir(r)
if err != nil { if err != nil {
return nil, err return FileBase{}, err
} }
return dir.Open(r, strings.Join(split[1:], "/")) return dir.Open(r, strings.Join(split[1:], "/"))
} }
+14 -14
View File
@@ -12,28 +12,28 @@ import (
) )
type FileBase struct { type FileBase struct {
Inode *inode.Inode Inode inode.Inode
Name string Name string
} }
func (r *Reader) BaseFromInode(i *inode.Inode, name string) *FileBase { func (r *Reader) BaseFromInode(i inode.Inode, name string) FileBase {
return &FileBase{Inode: i, Name: name} return FileBase{Inode: i, Name: name}
} }
func (r *Reader) BaseFromEntry(e directory.Entry) (*FileBase, error) { func (r *Reader) BaseFromEntry(e directory.Entry) (FileBase, error) {
in, err := r.InodeFromEntry(e) in, err := r.InodeFromEntry(e)
if err != nil { if err != nil {
return nil, err return FileBase{}, err
} }
return &FileBase{Inode: in, Name: e.Name}, nil return FileBase{Inode: in, Name: e.Name}, nil
} }
func (r *Reader) BaseFromRef(ref uint64, name string) (*FileBase, error) { func (r *Reader) BaseFromRef(ref uint64, name string) (FileBase, error) {
in, err := r.InodeFromRef(ref) in, err := r.InodeFromRef(ref)
if err != nil { if err != nil {
return nil, err return FileBase{}, err
} }
return &FileBase{Inode: in, Name: name}, nil return FileBase{Inode: in, Name: name}, nil
} }
func (b *FileBase) Uid(r *Reader) (uint32, error) { func (b *FileBase) Uid(r *Reader) (uint32, error) {
@@ -48,7 +48,7 @@ func (b *FileBase) IsDir() bool {
return b.Inode.Type == inode.Dir || b.Inode.Type == inode.EDir return b.Inode.Type == inode.Dir || b.Inode.Type == inode.EDir
} }
func (b *FileBase) ToDir(r *Reader) (*Directory, error) { func (b *FileBase) ToDir(r *Reader) (Directory, error) {
var blockStart uint32 var blockStart uint32
var size uint32 var size uint32
var offset uint16 var offset uint16
@@ -62,19 +62,19 @@ func (b *FileBase) ToDir(r *Reader) (*Directory, error) {
size = b.Inode.Data.(inode.EDirectory).Size size = b.Inode.Data.(inode.EDirectory).Size
offset = b.Inode.Data.(inode.EDirectory).Offset offset = b.Inode.Data.(inode.EDirectory).Offset
default: default:
return nil, errors.New("not a directory") return Directory{}, errors.New("not a directory")
} }
dirRdr := metadata.NewReader(toreader.NewReader(r.r, int64(r.Superblock.DirTableStart)+int64(blockStart)), r.d) dirRdr := metadata.NewReader(toreader.NewReader(r.r, int64(r.Superblock.DirTableStart)+int64(blockStart)), r.d)
defer dirRdr.Close() defer dirRdr.Close()
_, err := dirRdr.Read(make([]byte, offset)) _, err := dirRdr.Read(make([]byte, offset))
if err != nil { if err != nil {
return nil, err return Directory{}, err
} }
entries, err := directory.ReadDirectory(dirRdr, size) entries, err := directory.ReadDirectory(dirRdr, size)
if err != nil { if err != nil {
return nil, err return Directory{}, err
} }
return &Directory{ return Directory{
FileBase: *b, FileBase: *b,
Entries: entries, Entries: entries,
}, nil }, nil
+3 -3
View File
@@ -7,18 +7,18 @@ import (
"github.com/CalebQ42/squashfs/low/inode" "github.com/CalebQ42/squashfs/low/inode"
) )
func (r *Reader) InodeFromRef(ref uint64) (*inode.Inode, error) { func (r *Reader) InodeFromRef(ref uint64) (inode.Inode, error) {
offset, meta := (ref>>16)+r.Superblock.InodeTableStart, ref&0xFFFF offset, meta := (ref>>16)+r.Superblock.InodeTableStart, ref&0xFFFF
rdr := metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d) rdr := metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d)
defer rdr.Close() defer rdr.Close()
_, err := rdr.Read(make([]byte, meta)) _, err := rdr.Read(make([]byte, meta))
if err != nil { if err != nil {
return nil, err return inode.Inode{}, err
} }
return inode.Read(rdr, r.Superblock.BlockSize) return inode.Read(rdr, r.Superblock.BlockSize)
} }
func (r *Reader) InodeFromEntry(e directory.Entry) (*inode.Inode, error) { func (r *Reader) InodeFromEntry(e directory.Entry) (inode.Inode, error) {
rdr := metadata.NewReader(toreader.NewReader(r.r, int64(r.Superblock.InodeTableStart)+int64(e.BlockStart)), r.d) rdr := metadata.NewReader(toreader.NewReader(r.r, int64(r.Superblock.InodeTableStart)+int64(e.BlockStart)), r.d)
defer rdr.Close() defer rdr.Close()
rdr.Read(make([]byte, e.Offset)) rdr.Read(make([]byte, e.Offset))
+8 -15
View File
@@ -39,8 +39,7 @@ type Inode struct {
Data any Data any
} }
func Read(r io.Reader, blockSize uint32) (i *Inode, err error) { func Read(r io.Reader, blockSize uint32) (i Inode, err error) {
i = new(Inode)
err = binary.Read(r, binary.LittleEndian, &i.Header) err = binary.Read(r, binary.LittleEndian, &i.Header)
if err != nil { if err != nil {
return return
@@ -82,23 +81,17 @@ func Read(r io.Reader, blockSize uint32) (i *Inode, err error) {
func (i Inode) Mode() (out fs.FileMode) { func (i Inode) Mode() (out fs.FileMode) {
out = fs.FileMode(i.Perm) out = fs.FileMode(i.Perm)
switch i.Data.(type) { switch i.Type {
case Directory: case Dir, EDir:
out |= fs.ModeDir out |= fs.ModeDir
case EDirectory: case Sym, ESym:
out |= fs.ModeDir
case Symlink:
out |= fs.ModeSymlink out |= fs.ModeSymlink
case ESymlink: case Char, EChar, Block, EBlock:
out |= fs.ModeSymlink
case Device:
out |= fs.ModeDevice out |= fs.ModeDevice
case EDevice: case Fifo, EFifo:
out |= fs.ModeDevice
case IPC:
out |= fs.ModeNamedPipe
case EIPC:
out |= fs.ModeNamedPipe out |= fs.ModeNamedPipe
case Sock, ESock:
out |= fs.ModeSocket
} }
return return
} }
+18 -21
View File
@@ -32,7 +32,7 @@ var (
type Reader struct { type Reader struct {
r io.ReaderAt r io.ReaderAt
d decompress.Decompressor d decompress.Decompressor
Root *Directory Root Directory
fragTable []fragEntry fragTable []fragEntry
idTable []uint32 idTable []uint32
exportTable []uint64 exportTable []uint64
@@ -59,15 +59,21 @@ func NewReader(r io.ReaderAt) (rdr *Reader, err error) {
case ZlibCompression: case ZlibCompression:
rdr.d = decompress.Zlib{} rdr.d = decompress.Zlib{}
case LZMACompression: case LZMACompression:
rdr.d = decompress.Lzma{} rdr.d, err = decompress.NewLzma()
if err != nil {
return nil, err
}
case LZOCompression: case LZOCompression:
rdr.d = decompress.Lzo{} rdr.d, err = decompress.NewLzo()
if err != nil {
return nil, err
}
case XZCompression: case XZCompression:
rdr.d = decompress.Xz{} rdr.d = decompress.Xz{}
case LZ4Compression: case LZ4Compression:
rdr.d = decompress.Lz4{} rdr.d = decompress.Lz4{}
case ZSTDCompression: case ZSTDCompression:
rdr.d = &decompress.Zstd{} rdr.d = decompress.Zstd{}
default: default:
return nil, errors.New("invalid compression type. possible corrupted archive") return nil, errors.New("invalid compression type. possible corrupted archive")
} }
@@ -88,7 +94,7 @@ func (r *Reader) Id(i uint16) (uint32, error) {
// Populate the id table as needed // Populate the id table as needed
var blockNum uint32 var blockNum uint32
if i != 0 { // If i == 0, we go negatives causing issues with uint32s if i != 0 { // If i == 0, we go negatives causing issues with uint32s
blockNum = uint32(math.Ceil(float64(i)/2048)) - 1 blockNum = uint32(math.Ceil(float64(i+1)/2048)) - 1
} else { } else {
blockNum = 0 blockNum = 0
} }
@@ -105,10 +111,7 @@ func (r *Reader) Id(i uint16) (uint32, error) {
if err != nil { if err != nil {
return 0, err return 0, err
} }
idsToRead = r.Superblock.IdCount - uint16(len(r.idTable)) idsToRead = min(r.Superblock.IdCount-uint16(len(r.idTable)), 2048)
if idsToRead > 2048 {
idsToRead = 2048
}
idsTmp = make([]uint32, idsToRead) idsTmp = make([]uint32, idsToRead)
rdr = metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d) rdr = metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d)
err = binary.Read(rdr, binary.LittleEndian, &idsTmp) err = binary.Read(rdr, binary.LittleEndian, &idsTmp)
@@ -131,7 +134,7 @@ func (r *Reader) fragEntry(i uint32) (fragEntry, error) {
// Populate the fragment table as needed // Populate the fragment table as needed
var blockNum uint32 var blockNum uint32
if i != 0 { // If i == 0, we go negatives causing issues with uint32s if i != 0 { // If i == 0, we go negatives causing issues with uint32s
blockNum = uint32(math.Ceil(float64(i)/512)) - 1 blockNum = uint32(math.Ceil(float64(i+1)/512)) - 1
} else { } else {
blockNum = 0 blockNum = 0
} }
@@ -148,10 +151,7 @@ func (r *Reader) fragEntry(i uint32) (fragEntry, error) {
if err != nil { if err != nil {
return fragEntry{}, err return fragEntry{}, err
} }
fragsToRead = r.Superblock.FragCount - uint32(len(r.fragTable)) fragsToRead = min(r.Superblock.FragCount-uint32(len(r.fragTable)), 512)
if fragsToRead > 512 {
fragsToRead = 512
}
fragsTmp = make([]fragEntry, fragsToRead) fragsTmp = make([]fragEntry, fragsToRead)
rdr = metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d) rdr = metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d)
err = binary.Read(rdr, binary.LittleEndian, &fragsTmp) err = binary.Read(rdr, binary.LittleEndian, &fragsTmp)
@@ -177,7 +177,7 @@ func (r *Reader) inodeRef(i uint32) (uint64, error) {
// Populate the export table as needed // Populate the export table as needed
var blockNum uint32 var blockNum uint32
if i != 0 { // If i == 0, we go negatives causing issues with uint32s if i != 0 { // If i == 0, we go negatives causing issues with uint32s
blockNum = uint32(math.Ceil(float64(i)/1024)) - 1 blockNum = uint32(math.Ceil(float64(i+1)/1024)) - 1
} else { } else {
blockNum = 0 blockNum = 0
} }
@@ -194,10 +194,7 @@ func (r *Reader) inodeRef(i uint32) (uint64, error) {
if err != nil { if err != nil {
return 0, err return 0, err
} }
refsToRead = r.Superblock.InodeCount - uint32(len(r.exportTable)) refsToRead = min(r.Superblock.InodeCount-uint32(len(r.exportTable)), 1024)
if refsToRead > 1024 {
refsToRead = 1024
}
refsTmp = make([]uint64, refsToRead) refsTmp = make([]uint64, refsToRead)
rdr = metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d) rdr = metadata.NewReader(toreader.NewReader(r.r, int64(offset)), r.d)
err = binary.Read(rdr, binary.LittleEndian, &refsTmp) err = binary.Read(rdr, binary.LittleEndian, &refsTmp)
@@ -210,10 +207,10 @@ func (r *Reader) inodeRef(i uint32) (uint64, error) {
return r.exportTable[i], nil return r.exportTable[i], nil
} }
func (r *Reader) Inode(i uint32) (*inode.Inode, error) { func (r *Reader) Inode(i uint32) (inode.Inode, error) {
ref, err := r.inodeRef(i) ref, err := r.inodeRef(i)
if err != nil { if err != nil {
return nil, err return inode.Inode{}, err
} }
return r.InodeFromRef(ref) return r.InodeFromRef(ref)
} }
+23 -10
View File
@@ -1,4 +1,4 @@
package squashfslow_test package squashfslow
import ( import (
"fmt" "fmt"
@@ -8,13 +8,11 @@ import (
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"testing" "testing"
squashfslow "github.com/CalebQ42/squashfs/low"
) )
const ( const (
squashfsURL = "https://darkstorm.tech/files/LinuxPATest.sfs" squashfsURL = "https://darkstorm.tech/files/LinuxPATest.sfs"
squashfsName = "LinuxPATest.sfs" squashfsName = "airootfs.sfs"
) )
func preTest(dir string) (fil *os.File, err error) { func preTest(dir string) (fil *os.File, err error) {
@@ -50,6 +48,21 @@ func preTest(dir string) (fil *os.File, err error) {
return return
} }
func TestMisc(t *testing.T) {
tmpDir := "../testing"
fil, err := preTest(tmpDir)
if err != nil {
t.Fatal(err)
}
defer fil.Close()
rdr, err := NewReader(fil)
if err != nil {
t.Fatal(err)
}
t.Log(rdr.Superblock.FragCount)
t.Fatal(rdr.fragEntry(1233))
}
func TestReader(t *testing.T) { func TestReader(t *testing.T) {
tmpDir := "../testing" tmpDir := "../testing"
fil, err := preTest(tmpDir) fil, err := preTest(tmpDir)
@@ -57,7 +70,7 @@ func TestReader(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
defer fil.Close() defer fil.Close()
rdr, err := squashfslow.NewReader(fil) rdr, err := NewReader(fil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -77,7 +90,7 @@ func TestSingleFile(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
defer fil.Close() defer fil.Close()
rdr, err := squashfslow.NewReader(fil) rdr, err := NewReader(fil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -88,11 +101,11 @@ func TestSingleFile(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
err = extractToDir(rdr, b, path) err = extractToDir(rdr, &b, path)
t.Fatal(err) t.Fatal(err)
} }
func extractToDir(rdr *squashfslow.Reader, b *squashfslow.FileBase, folder string) error { func extractToDir(rdr *Reader, b *FileBase, folder string) error {
path := filepath.Join(folder, b.Name) path := filepath.Join(folder, b.Name)
if b.IsDir() { if b.IsDir() {
d, err := b.ToDir(rdr) d, err := b.ToDir(rdr)
@@ -103,13 +116,13 @@ func extractToDir(rdr *squashfslow.Reader, b *squashfslow.FileBase, folder strin
if err != nil { if err != nil {
return err return err
} }
var nestBast *squashfslow.FileBase var nestBast FileBase
for _, e := range d.Entries { for _, e := range d.Entries {
nestBast, err = rdr.BaseFromEntry(e) nestBast, err = rdr.BaseFromEntry(e)
if err != nil { if err != nil {
return err return err
} }
err = extractToDir(rdr, nestBast, path) err = extractToDir(rdr, &nestBast, path)
if err != nil { if err != nil {
return err return err
} }
+7 -2
View File
@@ -4,12 +4,13 @@ import (
"io" "io"
"time" "time"
"github.com/CalebQ42/squashfs/internal/toreader"
squashfslow "github.com/CalebQ42/squashfs/low" squashfslow "github.com/CalebQ42/squashfs/low"
) )
type Reader struct { type Reader struct {
*FS *FS
Low *squashfslow.Reader Low squashfslow.Reader
} }
func NewReader(r io.ReaderAt) (*Reader, error) { func NewReader(r io.ReaderAt) (*Reader, error) {
@@ -18,7 +19,7 @@ func NewReader(r io.ReaderAt) (*Reader, error) {
return nil, err return nil, err
} }
out := &Reader{ out := &Reader{
Low: rdr, Low: *rdr,
} }
out.FS = &FS{ out.FS = &FS{
d: rdr.Root, d: rdr.Root,
@@ -27,6 +28,10 @@ func NewReader(r io.ReaderAt) (*Reader, error) {
return out, nil return out, nil
} }
func NewReaderAtOffset(r io.ReaderAt, offset int64) (*Reader, error) {
return NewReader(toreader.NewOffsetReader(r, offset))
}
func (r *Reader) ModTime() time.Time { func (r *Reader) ModTime() time.Time {
return time.Unix(int64(r.Low.Superblock.ModTime), 0) return time.Unix(int64(r.Low.Superblock.ModTime), 0)
} }
+15 -15
View File
@@ -1,4 +1,4 @@
package squashfs_test package squashfs
//Actually proper tests go here. //Actually proper tests go here.
@@ -13,13 +13,11 @@ import (
"strconv" "strconv"
"testing" "testing"
"time" "time"
"github.com/CalebQ42/squashfs"
) )
const ( const (
squashfsURL = "https://darkstorm.tech/files/LinuxPATest.sfs" squashfsURL = "https://darkstorm.tech/files/LinuxPATest.sfs"
squashfsName = "airootfs.sfs" squashfsName = "LinuxPATest.sfs"
) )
func preTest(dir string) (fil *os.File, err error) { func preTest(dir string) (fil *os.File, err error) {
@@ -61,13 +59,13 @@ func TestMisc(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
rdr, err := squashfs.NewReader(fil) rdr, err := NewReader(fil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
_ = rdr _ = rdr
// Put testing here // Put testing here
t.Fatal("UM") // t.Fatal("UM")
} }
func BenchmarkRace(b *testing.B) { func BenchmarkRace(b *testing.B) {
@@ -81,9 +79,10 @@ func BenchmarkRace(b *testing.B) {
os.RemoveAll(libPath) os.RemoveAll(libPath)
os.RemoveAll(unsquashPath) os.RemoveAll(unsquashPath)
var libTime, unsquashTime time.Duration var libTime, unsquashTime time.Duration
op := squashfs.FastOptions() op := FastOptions()
op.IgnorePerm = true
start := time.Now() start := time.Now()
rdr, err := squashfs.NewReader(fil) rdr, err := NewReader(fil)
if err != nil { if err != nil {
b.Fatal(err) b.Fatal(err)
} }
@@ -109,7 +108,7 @@ func BenchmarkRace(b *testing.B) {
func TestExtractQuick(t *testing.T) { func TestExtractQuick(t *testing.T) {
//First, setup everything and extract the archive using the library and unsquashfs //First, setup everything and extract the archive using the library and unsquashfs
// tmpDir := b.TempDir() // tmpDir := bTempDir()
tmpDir := "testing" tmpDir := "testing"
fil, err := preTest(tmpDir) fil, err := preTest(tmpDir)
if err != nil { if err != nil {
@@ -119,13 +118,13 @@ func TestExtractQuick(t *testing.T) {
unsquashPath := filepath.Join(tmpDir, "ExtractSquashfs") unsquashPath := filepath.Join(tmpDir, "ExtractSquashfs")
os.RemoveAll(libPath) os.RemoveAll(libPath)
os.RemoveAll(unsquashPath) os.RemoveAll(unsquashPath)
rdr, err := squashfs.NewReader(fil) rdr, err := NewReader(fil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
os.RemoveAll(filepath.Join(tmpDir, "testLog.txt")) os.RemoveAll(filepath.Join(tmpDir, "testLog.txt"))
logFil, _ := os.Create(filepath.Join(tmpDir, "testLog.txt")) logFil, _ := os.Create(filepath.Join(tmpDir, "testLog.txt"))
op := squashfs.DefaultOptions() op := FastOptions()
op.Verbose = true op.Verbose = true
op.IgnorePerm = true op.IgnorePerm = true
op.LogOutput = logFil op.LogOutput = logFil
@@ -168,7 +167,7 @@ func TestExtractQuick(t *testing.T) {
} }
} }
var filePath = "bin" var filePath = "Start.exe"
func TestSingleFile(t *testing.T) { func TestSingleFile(t *testing.T) {
tmpDir := "testing" tmpDir := "testing"
@@ -177,7 +176,7 @@ func TestSingleFile(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
os.Remove(filepath.Join(tmpDir, filePath)) os.Remove(filepath.Join(tmpDir, filePath))
rdr, err := squashfs.NewReader(fil) rdr, err := NewReader(fil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -185,9 +184,10 @@ func TestSingleFile(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
err = f.(*squashfs.File).ExtractWithOptions("testing", &squashfs.ExtractionOptions{Verbose: true}) op := DefaultOptions()
op.Verbose = true
err = f.(*File).ExtractWithOptions("testing", op)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
t.Fatal("HI")
} }