Commit graph

3717 commits

Author SHA1 Message Date
Mike Fleetwood b7dd5480cd Remove unused udevinfo from README
Missed removing mention of udevinfo from README when this earlier commit
removed it's use by GParted.
    3f15a66291
    Drop use of long ago removed udevinfo
2020-02-26 16:41:43 +00:00
Mike Fleetwood 8ae9abada4 Wait for udev change on /dev/DISK when erasing signatures (#83)
A user reported that formatting a whole disk device with a file system
failed like this:

    Format /dev/sdd as ext4                                    (ERROR)
    + calibrate /dev/sdd                                       (SUCCESS)
        path: /dev/sdd (device)
        start: 0
        end: 15633407
        size: 15633408 (7.45 GiB)
    + clear old file system signatures in /dev/sdd             (SUCCESS)
      + write 512.00 KiB of zeros at byte offset 0             (SUCCESS)
      + write 4.00 KiB of zeros at byte offset 67108864        (SUCCESS)
      + write 512.00 KiB of zeros at byte offset 8003780608    (SUCCESS)
      + write 4.00 KiB of zeros at byte offset 8004239360      (SUCCESS)
      + write 8.00 KiB of zeros at byte offset 8004296704      (SUCCESS)
      + flush operating system cache of /dev/sdd               (SUCCESS)
    + create new ext4 file system                              (ERROR)
      + mkfs.ext4 -F -O ^64bit -L '' '/dev/sdd'                (ERROR)
        mke2fs 1.44.1 (24-Mar-2018)
        /dev/sdd is apparently in use by the system; will not make a filesystem here!

Opening the whole disk block device exclusively causes mkfs.ext4 to
report that error like this:

    # python
    >>> import os
    >>> f = os.open('/dev/sdb',os.O_RDONLY|os.O_EXCL)
    >>> ^Z
    [1]+  Stopped                 python
    # mkfs.ext4 -F -O ^64bit -L '' '/dev/sdb'
    mke2fs 1.42.9 (28-Dec-2013)
    /dev/sdb is apparently in use by the system; will not make a filesystem here!
    # echo $?
    1

I have not been able to reproduce this error, but with debugging and
sleeping in GParted, stracing GParted and using 'udevadm monitor' to
watch udev events the following sequence of events is seen:

  gparted    |format(partition, operationdetail)
  gparted    |  erase_filesystem_signatures(partition, operationdetail)
  gparted    |    get_device(device_path="/dev/sdb", lp_device, flush=false)
  gparted    |      ped_device_get("/dev/sdb")
  libparted  |        open("/dev/sdb", O_RDONLY) = 11
  libparted  |        close(11)
  gparted    |    ped_device_open(lp_device)
  libparted  |      open("/dev/sdb", O_RDWR) = 11
  gparted    |    ped_device_sync(lp_device)
  libparted  |      ioctl(11, BLKFLSBUF)
  gparted    |    ped_device_close()
  libparted  |      close(11)
  udev(async)|        KERNEL change /devices/.../sdb (block)
  udev(async)|        UDEV   change /devices/.../sdb (block)
  gparted    |  set_partition_type(partition, operationdetail)
  gparted    |  create_filesystem(partition, operationdetail)
  gparted    |    ext2::create(partition, operationdetail)
  gparted    |      FileSystem::execute_command("mkfs.ext4 -F -O ^64bit -L '' '/dev/sdb')

So it is assumed that the processing of the udev change rule after
closing the block device in erase_filesystem_signatures() overlaps with
the execution mkfs.ext4 and causes the seen error.  Fix by waiting for
those udev events to complete as was previously done by commits [1][2]
[3].

Also note that this is specific to creating file systems on and
formatting unpartitioned whole disk devices because set_partition_type()
is a no-operation.  Where as on a partitioned device
set_partition_type() calls commit() which already waits for udev rules
to complete [3].

[1] 50c8924a8e4d9cc96a2ea45f13291114402affee
    Wait for udev to recreate /dev/PTN entries when querying partition
    FSs (!46)
[2] 4f6c312e3bc68cafb5e6035fd4a5b5bbbfcea992
    Wait for udev change on /dev/DISK when querying whole device FS
    (!46)
[3] 2f53876c0f
    Wait for the kernel and udev to settle partitions for a second time
    (#790418)

Closes #83 - /dev/sdd is apparently in use by the system; will not make
             a filesystem here!
2020-02-26 16:41:43 +00:00
Daniele Forsi 24c0b81bfc Fix formatting directive in it.po (!58)
Fixes:
    (gpartedbin:78003): glibmm-WARNING **: 22:55:05.465: invalid
    substitution "%s" in fmt string "percorso: %1 (%s)"

Closes !58 - Fix formatting directive in it.po (!58)
2020-02-26 13:37:33 +00:00
Nathan Follens 2f9af89545 Update Dutch translation 2020-02-26 13:28:47 +00:00
Daniel Korostil c559c8869d Update Ukrainian translation 2020-02-23 15:06:19 +00:00
Daniel Mustieles 2b0666c7a7 Updated Spanish translation 2020-02-04 15:45:50 +01:00
Dušan Kazik 1a8fd5ff0a Update Slovak translation 2020-01-29 13:38:39 +00:00
Curtis Gedak d653fe6173 Append -git to version for continuing development
Also fix minor whitespace mistake in NEWS
2020-01-20 10:33:47 -07:00
Curtis Gedak 8a15021168 ========== gparted-1.1.0 ========== 2020-01-20 10:15:32 -07:00
Curtis Gedak 77de664881 Update copyright years 2020-01-20 09:56:07 -07:00
A S Alam f7a12b624a Update Punjabi translation 2020-01-19 17:23:15 +00:00
Rūdolfs Mazurs b7dba0dca7 Update Latvian translation 2020-01-15 18:57:16 +00:00
Kukuh Syafaat acc92ca1f9 Update Indonesian translation 2020-01-14 15:08:06 +00:00
Bruce Cowan df285fbd7a Update British English translation 2020-01-12 21:10:09 +00:00
Wolfgang Stöggl 2e4961b9be Update German translation 2020-01-12 18:31:31 +00:00
Claude Paroz d871a5395e Updated French translation 2020-01-11 10:25:53 +01:00
Trần Ngọc Quân 18c787c2f0 Updated Vietnamese translation
Signed-off-by: Trần Ngọc Quân <vnwildman@gmail.com>
2020-01-11 14:19:52 +07:00
Sveinn í Felli a6db989a9b Update Icelandic translation 2020-01-10 18:25:40 +00:00
Mike Fleetwood ce18685dfa Save all files on CI job failure for investigation
Since patchset !49 "Add file system interface tests" was merged the
GitLab Continuous Integration jobs have sometimes failed executing
test_SupportedFilesystems.  So on failure save all files from the docker
image for 1 week for investigation.

Documentation:
* Introduction to job artifacts
  https://gitlab.gnome.org/help/user/project/pipelines/job_artifacts.md

* GitLab CI/CD Pipeline Configuration Reference, artifacts
  https://gitlab.gnome.org/help/ci/yaml/README.md#artifacts
2019-12-04 07:38:01 +00:00
Mike Fleetwood 01826277e3 Rename TreeView_Detail::treeview_filesystems_Columns member to fsname (!52)
Closes !52 - Rename members and variables currently named 'filesystem'
2019-12-04 07:38:01 +00:00
Mike Fleetwood e5b92a87f2 Rename DialogFeatures::treeview_filesystems_Columns member to fsname (!52)
Name the structure member to 'fsname' used to store strings like "ext2"
etc.  This is equivalent to what was previously done in this commit:
    a9f08ddc7d
    Rename local variable to fsname in get_filesystem() (#741430)

Closes !52 - Rename members and variables currently named 'filesystem'
2019-12-04 07:38:01 +00:00
Mike Fleetwood f53c462c06 Rename create_format_menu_add_item() parameter of type FSType (!52)
Closes !52 - Rename members and variables currently named 'filesystem'
2019-12-04 07:38:00 +00:00
Mike Fleetwood 2cbb867508 Rename set_device_partitions() local variable of type FSType (!52)
Closes !52 - Rename members and variables currently named 'filesystem'
2019-12-04 07:38:00 +00:00
Mike Fleetwood c85fc66dcf Rename Utils method parameters of type FSType (!52)
Closes !52 - Rename members and variables currently named 'filesystem'
2019-12-04 07:38:00 +00:00
Mike Fleetwood 58fb230fb0 Also rename FS.filesystem member to fstype (!52)
Closes !52 - Rename members and variables currently named 'filesystem'
2019-12-04 07:37:19 +00:00
Mike Fleetwood b0f92be638 Rename Partition.filesystem member to fstype (!52)
Previously made this change:
    175d27c55d
    Rename enum FILESYSTEM to FSType

Now complete the renaming exercise of members and variables currently
named 'filesystem'.

Closes !52 - Rename members and variables currently named 'filesystem'
2019-12-03 13:24:44 +00:00
Mike Fleetwood 047a2481bb Stop requesting partition paths of free space and metadata
In GParted_Core::set_device_partitions() the partition path is being
queried from libparted.  However this is done before the switch
statement on the type of the partition, so is called for all libparted
partition objects including PED_PARTITION_FREESPACE and
PED_PARTITION_METADATA ones.  As libparted numbers these partition
objects as -1, it returns paths like "/dev/sda-1".

Additionally when using GParted, with it's default DMRaid handling, on a
dmraid started array this results in paths like
"/dev/mapper/isw_ecccdhhiga_MyArray-1" being passed to
is_dmraid_device() and make_path_dmraid_compatible().  Fortunately
make_path_dmraid_compatible() does nothing and returns the same name.
Call chain looks like:

    GParted_Core::set_device_partitions()
      get_partition_path(lp_partition)
        // where:
        // lp_partition->disk->dev->path = "/dev/mapper/isw_ecccdhhiga_MyArray"
        // lp_partition->type == PED_PARTITION_FREESPACE |
        //                       PED_PARTITION_METADATA
        //              ->num == -1
        ped_partition_get_path(lp_partition)
          return "/dev/mapper/isw_ecccdhhiga_MyArray-1"
        dmraid.is_dmraid_supported()
        dmraid.is_dmraid_device("/dev/mapper/isw_ecccdhhiga_MyArray-1")
          return true
        dmraid.make_path_dmraid_compatible("/dev/mapper/isw_ecccdhhiga_MyArray-1")
          return "/dev/mapper/isw_ecccdhhiga_MyArray-1"

Fix by moving the get_partition_path() call inside the switch statement
so that it is only called for PED_PARTITION_NORMAL,
PED_PARTITION_LOGICAL and PED_PARTITION_EXTENDED partition types.

Relevant commits:
*   53c49349f7
    Simplify logic in set_device_partitions method

*   81986c0990
    Ensure partition path name is compatible with dmraid (#622217)
2019-12-02 16:35:22 +00:00
Mike Fleetwood fa682d372a Make 4 internally used only DMRaid methods private 2019-12-02 16:35:22 +00:00
Mike Fleetwood 21cad97dc7 Recognise ATARAID members started by dmraid (#75)
This is not strictly necessary as members are already recognised using
blkid since this commit earlier in the sequence "Recognise ATARAID
members (#75)".  However it makes sure active members are recognised
even if blkid is not available and matches how file system detection
queries the SWRaid_Info module.

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood bb865aaaa4 Display array device as mount point of dmraid started ATARAID members (#75)
This matches how the array device is displayed as the mount point for
mdadm started ATARAID members by "Display array device as mount point of
mdadm started ATARAID members (#75)" earlier in this patchset.

Extend the DMRaid module member cache to save the array device name and
use as needed to display as the mount point.

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood caec22871e Detect busy status of dmraid started ATARAID members (#75)
Again this is to stop GParted allowing overwrite operations being
performed on an ATARAID member while the array is actively using the
member.  This time for dmraid started arrays using the kernel DM (Device
Mapper) driver.

The DMRaid module already uses dmraid to report active array names:

    # dmraid -sa -c
    isw_ecccdhhiga_MyArray

To find active members in this array, (1) use udev to lookup the kernel
device name:

    # udevadm info --query=name /dev/mapper/isw_ecccdhhiga_MyArray
    dm-0

(2) list the member names exposed by the kernel DM driver through the
/sys file system.

    # ls /sys/block/dm-0/slaves
    sdc  sdd
    # ls -l /sys/block/dm-0/slaves
    lrwxrwxrwx 1 root root 0 Nov 24 09:52 sdc -> ../../../../pci0000:00/0000:00:0d.0/ata3/host2/target2:0:0/2:0:0:0/block/sdc
    lrwxrwxrwx 1 root root 0 Nov 24 09:52 sdc -> ../../../../pci0000:00/0000:00:0d.0/ata4/host3/target3:0:0/3:0:0:0/block/sdd

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood 425dfa3709 Enable basic supported actions for ATARAID members (#75)
When an ATARAID member is inactive allow basic supported actions of
copy and move to be performed like with other recognised but only basic
supported types.

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood 1f1f44ff7a Prevent unmount of busy ATARAID members (#75)
Since earlier commit "Display array device as mount point of mdadm
started ATARAID members (#75)" GParted allows attempting to unmout a
busy ATARAID member as if it was a file system.  This is not a valid
thing to do, so disallow it.

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood f6c86835eb Display array uuid of mdadm recognised ATARAID members (#75)
Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood 538c866d09 Display array device as mount point of mdadm started ATARAID members (#75)
This matches how other non-file systems are handled, by displaying the
access reference in the mount point column.  For LVM Physical Volumes
the Volume Group name is displayed [1] and for an active Linux Software
RAID array the array device is displayed [2].

[1] 8083f11d84
    Display LVM2 VGNAME as the PV's mount point (#160787)

[2] f6c2f00df7
    Populate member mount point with SWRaid array device (#756829)

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood 6e990ea48a Detect busy status of mdadm started ATARAID members (#75)
This stops GParted allowing overwrite operations (such as create
partition table or format with a whole device file system) being
performed on an ATARAID member while the array is actively using the
member.

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood ef6794b7de Display correct type of mdadm recognised ATARAID members (#75)
The previous commit, made mdadm recognised IMSM and DDF type ATARAID
members get displayed as "linux-raid" (Linux Software RAID array
member).  This was because of query method 1 in detect_filesystems().

Fix this now by exposing and using the fstype of the member from the
SWRaid_Info cache.

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood 73bf8bef62 Parse ATARAID members from mdadm output and /proc/mdstat (#75)
Since mdadm release 3.0 (2009-06-02) [1] it has also supported external
metadata formats IMSM (Intel Matrix Storage Manager) and DDF, previously
only managed by dmraid.

A number of distributions have switched to use mdadm and kernel MD
(Multiple Devices) driver for managing these Firmware / BIOS / ATARAID
arrays.  These include: Fedora >= 14 [2], RHEL / CentOS >= 6 [3],
SLES >= 12 [4], Ubuntu >= 16.04 LTS.

Therefore additionally parse members in these ATARAID arrays included in
mdadm output, and when activated using the kernel MD driver, in file
/proc/mdstat.  Add fstype to the SWRaid_Info cache records to
distinguish members apart.  So far the rest of the GParted code
continues to treat all members as FS_LINUX_SWRAID.  This will be
resolved in following commits.

Note that this in no way affects how GParted shows and partitions the
array device itself, even those managed by dmraid and use the GParted
DMRaid module.  It only affects how GParted shows the member drives
themselves.

[1] mdadm ANNOUNCE-3.0 file
    https://git.kernel.org/pub/scm/utils/mdadm/mdadm.git/tree/ANNOUNCE-3.0?h=mdadm-3.0

[2] Fedora 14, Storage Administration Guide, 12.5. Linux RAID Subsystem
    https://docs.fedoraproject.org/en-US/Fedora/14/html/Storage_Administration_Guide/raid-subsys.html
    "...  Fedora 14 uses mdraid with external metadata to access ISW /
    IMSM (Intel firmware RAID) sets.  mdraid sets are configured and
    controlled through the mdadm utility."

[3] RHEL 6, Storage Administration Guide, 17.3. Linux RAID Subsystem
    https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/raid-subsys
    "mdraid also supports other metadata formats, known as external
    metadata.  Red Hat Enterprise Linux 6 uses mdraid with external
    metadata to access ISW / IMSM (Intel firmware RAID) sets.  mdraid
    sets are configured and controlled through the mdadm utility."

[4] SUSE Linux Enterprise Server 12 Release Notes, 7.2.3 Driver for IMSM
    and DDF
    https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12/#fate-316007
    "For IMSM and DDF RAIDs the mdadm driver is used unconditionally."

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Mike Fleetwood aea6200d5f Recognise ATARAID members (#75)
PATCHSET OVERVIEW

A user had a Firmware / BIOS / ATARAID array of 2 devices configured as
a RAID 0 (stripe) set.  On top of that was a GPT with the OS partitions.
GParted displays the following errors on initial load and subsequent
refresh:

        Libparted Error
    (-) Invalid argument during seek for read on /dev/sda
                          [ Retry ] [ Cancel ] [ Ignore ]

        Libparted Error
    (-) The backup GPT table is corrupt, but the
        primary appears OK, so that will be used.
                              [  Ok  ] [ Cancel ]

This is an Intel Software RAID array which stores metadata at the end of
each member device, and so the first 128 KiB stripe of the set is stored
in the first 128 KiB of the first member device /dev/sda which includes
the GPT for the whole RAID 0 device.  Hence when libparted reads member
device /dev/sda it finds a GPT describing a block device twice it's
size and in results the above errors when trying to read the backup GPT.

A more dangerous scenario occurs when using 2 devices configured in an
Intel Software RAID 1 (mirrored) set with GPT on top.  On refresh
GParted display this error for both members, /dev/sda and /dev/sdb:

        Libparted Warning
    /!\ Not all of the space available to /dev/sda appears to be used,
        you can fix the GPT to use all of the space (an extra 9554
        blocks) or continue with the current setting?
                                                  [  Fix  ] [ Ignore ]

Selecting [Fix] gets libparted to re-write the backup GPT to the end of
the member device, overwriting the ISW metadata!  Do that twice and both
copies of the metadata are gone!

Worked example of this more dangerous mirrored set case.  Initial setup:

    # dmraid -s
    *** Group superset isw_caffbiaegi
    --> Subset
    name   : isw_caffbiaegi_MyMirror
    size   : 16768000
    stride : 128
    type   : mirror
    status : ok
    subsets: 0
    devs   : 2
    spares : 0

    # dmraid -r
    /dev/sda: isw, "isw_caffbiaegi", GROUP, ok, 16777214 sectors, data@ 0
    /dev/sdb: isw, "isw_caffbiaegi", GROUP, ok, 16777214 sectors, data@ 0

    # wipefs /dev/sda
    offset               type
    ---------------------------------------------
    0x200                gpt   [partition table]
    0x1fffffc00          isw_raid_member   [raid]

Run GParted and click [Fix] on /dev/sda.  Now the first member has gone:

    # dmraid -s
    *** Group superset isw_caffbiaegi
    --> *Inconsistent* Subset
    name   : isw_caffbiaegi_MyMirror
    size   : 16768000
    stride : 128
    type   : mirror
    status : inconsistent
    subsets: 0
    devs   : 1
    spares : 0

    # dmraid -r
    /dev/sdb: isw, "isw_caffbiaegi", GROUP, ok, 16777214 sectors, data@ 0

    # wipefs /dev/sda
    offset               type
    ---------------------------------------------
    0x200                gpt   [partition table]

Click [Fix] on /dev/sdb.  Now all members of the array are gone:

    # dmraid -s
    no raid disks

    # dmraid -r
    no raid disks

    # wipefs /dev/sdb
    offset               type
    ---------------------------------------------
    0x200                gpt   [partition table]

So GParted must not run libparted partition table scanning on the member
devices in ATARAID arrays.  Only on the array device itself.

In terms of the UI GParted must show disks which are ATARAID members as
whole disk devices with ATARAID member content and detect array busy
status to avoid allowing active members from being overwritten while in
use.

THIS COMMIT

Recognise ATARAID member devices and display in GParted as whole device
"ataraid" file systems.  Because they are recognised as whole device
content ("ataraid" file systems) this alone stops GParted running the
libparted partition table scanning and avoids the above errors.

The list of dmraid supported formats is matched by the signatures
recognised by blkid:

    $ dmraid -l
    asr     : Adaptec HostRAID ASR (0,1,10)
    ddf1    : SNIA DDF1 (0,1,4,5,linear)
    hpt37x  : Highpoint HPT37X (S,0,1,10,01)
    hpt45x  : Highpoint HPT45X (S,0,1,10)
    isw     : Intel Software RAID (0,1,5,01)
    jmicron : JMicron ATARAID (S,0,1)
    lsi     : LSI Logic MegaRAID (0,1,10)
    nvidia  : NVidia RAID (S,0,1,10,5)
    pdc     : Promise FastTrack (S,0,1,10)
    sil     : Silicon Image(tm) Medley(tm) (0,1,10)
    via     : VIA Software RAID (S,0,1,10)
    dos     : DOS partitions on SW RAIDs

    $ fgrep -h _raid_member util-linux/libblkid/src/superblocks/*.c
            .name           = "adaptec_raid_member",
            .name           = "ddf_raid_member",
            .name           = "hpt45x_raid_member",
            .name           = "hpt37x_raid_member",
            .name           = "isw_raid_member",
            .name           = "jmicron_raid_member",
            .name           = "linux_raid_member",
            .name           = "lsi_mega_raid_member",
            .name           = "nvidia_raid_member",
            .name           = "promise_fasttrack_raid_member",
            .name           = "silicon_medley_raid_member",
            .name           = "via_raid_member",

As they are all types of Firmware / BIOS / ATARAID arrays, report all
members as a single "ataraid" file system type.  (Except for
"linux_raid_member" in the above blkid source listing which is Linux
Software RAID).

Closes #75 - Errors with GPT on RAID 0 ATARAID array
2019-12-02 16:35:22 +00:00
Andre Klapper 79165d6918 Fix an incorrect translation in the Spanish user help
See #80
2019-12-01 17:44:46 +01:00
Andre Klapper 5def6be7a2 Fix typo in Spanish translation of user help
Fixes #80
2019-12-01 13:55:45 +01:00
Daniel Mustieles eadc85dae1 Updated Spanish translation 2019-11-28 13:48:41 +01:00
Mike Fleetwood af60f91f7f Add missing includes into jfs.cc 2019-11-14 17:12:06 +00:00
Mike Fleetwood 4b8d4be789 Remove unallocated space comment from HACKING file (!50)
The HACKING file should be hints for making changes to the code base and
associated processes.  A overview of how GParted handled unallocated
space was not that.  Also now the size of a JFS is accurately calculated
using JFS as an example of a file system with intrinsic unallocated
space is no longer valid.  Therefore removed from the HACKING file.
Instead add the original commit message as an extended comment to method
calc_significant_unallocated_sectors().

Closes !50 - Calculate JFS size accurately
2019-11-14 17:12:06 +00:00
Mike Fleetwood 2c0572e296 Calculate mounted JFS size accurately (!50)
With the same minimum sized 16 MiB JFS used in the previous commit, now
mounted, GParted once again reports 1.20 MiB of unallocated space.  This
is because the kernel JFS driver is also just reporting the size of the
Aggregate Disk Map (dmap) as the size of the file system [1].

Fix by reading the on disk JFS superblock to calculate the size of the
file system, but query the free space from the kernel using statvfs().
Need to query mounted JFS free space from the kernel because the on disk
dmap is not updated immediately so doesn't reflect recently used or
freed disk space.

For example, start with the 16 MiB JFS empty and mounted.

    # echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
    [2] dn_nfree:           0x00000000eaa   [10] dn_agwidth:        1
    # df -k /mnt/1
    Filesystem     1K-blocks  Used Available Use% Mounted on
    /dev/sdb1          15152   136     15016   1% /mnt/1

Write 10 MiB of data to it:

    # dd if=/dev/zero bs=1M count=10 of=/mnt/1/file_10M
    10+0 records in
    10+0 records out
    1048760 bytes (10 MB, 10 MiB) copied, 0.0415676 s, 252 MB/s

Query the file system free space from the kernel and by reading the on
disk dmap figure:

    # df -k /mnt/1
    Filesystem     1K-blocks  Used Available Use% Mounted on
    /dev/sdb1          15152 10376      4776  69% /mnt/1
    # echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
    [2] dn_nfree:           0x00000000eaa   [10] dn_agwidth:        1

    # sync
    # echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
    [2] dn_nfree:           0x00000000eaa   [10] dn_agwidth:        1

    # umount /mnt/1
    # echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1 | fgrep dn_nfree
    [2] dn_nfree:           0x000000004aa   [10] dn_agwidth:        1

The kernel reports the updated usage straight away, but the on disk dmap
record doesn't get updated even by sync, only after unmounting.

This is the same fix as was previously done for EXT2/3/4 [2].

[1] Linux jfs_statfs() function
    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/jfs/super.c?h=v3.10#n142

[2] 3828019030
    Read file system size for mounted ext2/3/4 from superblock (#683255)

Closes !50 - Calculate JFS size accurately
2019-11-14 17:12:06 +00:00
Mike Fleetwood e55d10b919 Calculate unmounted JFS size accurately (!50)
Create the smallest possible JFS (16 MiB) and GParted will report
1.2 MiB of unallocated space.  This is because the size of the Aggregate
Disk Map (dmap) was used as the size of the file system.  However after
reading the source code to mkfs.jfs, it separately accounts for the size
of the Log (Journal) and the FSCK Working Space.  The size of a JFS is
the sum of these 3 components added together.

Using the minimum 16 MiB JFS as an example:

    # jfs_debugfs /dev/sdb1
    jfs_debugfs version 1.1.15, 04-Mar-2011

    Aggregate Block Size: 4096

    > superblock
    [1] s_magic:            'JFS1'          [15] s_ait2.addr1:      0x00
    [2] s_version:          1               [16] s_ait2.addr2:      0x00000018
    [3] s_size:     0x0000000000007660           s_ait2.address:    24
    [4] s_bsize:            4096            [17] s_logdev:          0x00000000
    [5] s_l2bsize:          12              [18] s_logserial:       0x00000000
    [6] s_l2bfactor:        3               [19] s_logpxd.len:      256
    [7] s_pbsize:           512             [20] s_logpxd.addr1:    0x00
    [8] s_l2pbsize:         9               [21] s_logpxd.addr2:    0x00000f00
    [9] pad:                Not Displayed        s_logpxd.address:  3840
    [10] s_agsize:          0x00002000      [22] s_fsckpxd.len:     52
    [11] s_flag:            0x10200900      [23] s_fsckpxd.addr1:   0x00
                            JFS_LINUX       [24] s_fsckpxd.addr2:   0x00000ecc
            JFS_COMMIT      JFS_GROUPCOMMIT      s_fsckpxd.address: 3788
                            JFS_INLINELOG   [25] s_time.tv_sec:     0x5dbbdfa0
                                            [26] s_time.tv_nsec:    0x00000000
                                            [27] s_fpack:           'small_jfs'
    [12] s_state:           0x00000000
                 FM_CLEAN
    [13] s_compress:        0
    [14] s_ait2.len:        4

    display_super: [m]odify or e[x]it: x
    > dmap

    Block allocation map control page at block 16

    [1] dn_mapsize:         0x00000000ecc   [9] dn_agheigth:        0
    [2] dn_nfree:           0x00000000eaa   [10] dn_agwidth:        1
    [3] dn_l2nbperpage:     0               [11] dn_agstart:        341
    [4] dn_numag:           1               [12] dn_agl2size:       13
    [5] dn_maxlevel:        0               [13] dn_agfree:         type 'f'
    [6] dn_maxag:           0               [14] dn_agsize:         8192
    [7] dn_agpref:          0               [15] pad:               Not Displayed
    [8] dn_aglevel:         0
    display_dbmap: [m]odify, [f]ree count, [t]ree, e[x]it > x
    > quit

Values of interest:
    s_size        - Aggregate size in device (s_pbsize) blocks
    s_bsize       - Aggregate block (aka file system allocation) size in
                    bytes
    s_pbsize      - Physical (device) block size in bytes
    s_logpxd.len  - Log (Journal) size in Aggregate (s_bsize) blocks
    s_fsckpxd.len - FSCK Working Space in Aggregate (s_bsize) blocks
    dn_nfree      - Number of free (s_bsize) blocks in Aggregate

Calculation:
    file system size = s_size * s_pbsize
                     + s_logpxd.len * s_bsize
                     + s_fsckpxd.len * s_bsize
                     = 30304 * 512
                     + 256 * 4096
                     + 52 * 4096
                     =  16777216
                        (Exactly 16 MiB.  The size of the partition.)
    free space = dn_nfree * s_bsize
               = 3754 * 4096
               = 15376384

Rewrite JFS usage querying code to use this updated calculation.

[1] JFS Overview / How the Journaled File System cuts system restart
    times to the quick
    http://jfs.sourceforge.net/project/pub/jfs.pdf
[2] JFS Layout / How the Journaled File systems handles the on-disk
    layout
    http://jfs.sourceforge.net/project/pub/jfslayout.pdf
[3] mkfs.jfs source code
    http://jfs.sourceforge.net/project/pub/jfsutils-1.1.15.tar.gz
    mkfs/mkfs.c
    Selected lines from mkfs/mkfs.c
        create_aggregate(..., number_of_blocks, ..., logsize, ...)
            number_of_blocks -= fsck_wspace_length;
            aggr_superblock.s_size = number_of_blocks * (aggr_block_size / phys_block_size);
            aggr_superblock.s_bsize = aggr_block_size;
            aggr_superblock.s_pbsize = phys_block_size;
            PXDlength(&aggr_superblock.s_logpxd, logsize);
            PXDlength(&aggr_superblock.s_fsckpxd, fsck_wspace_length);
        main()
            number_of_bytes = bytes_on_device;
            number_of_blocks = number_of_bytes / agg_block_size;
            logsize = logsize_in_bytes / aggr_block_size;
            number_of_blocks -= logsize;
            create_aggregate(..., number_of_blocks, ..., logsize, ...);

Closes !50 - Calculate JFS size accurately
2019-11-14 17:12:06 +00:00
Mike Fleetwood 7be6d0967a Update name of the NILFS2 specific package
Upstream NILFS project calls the package nilfs-utils [1][2].  Arch Linux
/ CentOS / Fedora / OpenSUSE use the upstream name.  However Debian /
Ubuntu name it nilfs-tools [3] instead.

Document the needed software as:

    nilfs-utils / nilfs-tools

Upstream name first separated by slash from alternative names
distributions use.

[1] NILFS Download page
    https://nilfs.sourceforge.io/en/download.html
[2] NILFS Public Git Repositories
    https://nilfs.sourceforge.io/en/git_repos.html
[3] Debian package: nilfs-tools
    https://packages.debian.org/sid/nilfs-tools
2019-11-09 17:18:34 +00:00
Mike Fleetwood 530e84bace Add missing libuuid-devel build dependency for Fedora into README 2019-11-09 17:18:34 +00:00
Mike Fleetwood 7a7d0a2119 Avoid crash reading JFS usage on Fedora 30 (!49)(#794947)
Running JFS read usage test on Fedora 30 fails like this:

    $ ./test_SupportedFileSystems --gtest_filter='*ReadUsage/jfs'
...
    [ RUN      ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs
    unknown file: Failure
    C++ exception with description "std::bad_alloc" thrown in the test body.
    [  FAILED  ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs, where GetParam() = 17 (41833 ms)

However the same test passes on Fedora 29, Fedora 31 Beta, CentOS 7,
Debian 10 and Ubuntu 18.04 LTS.

Also running GParted on Fedora 30 crashes just the same when reading JFS
usage:

    # gparted
    GParted 1.0.0
    configuration --enable-libparted-dmraid --enable-online-resize
    libparted 3.2
    terminate called after throwing an instance of 'std::bad_alloc'
      what():  std::bad_alloc
    /usr/bin/gparted: line 202: 19218 Aborted                 (core dumped) $BASE_CMD

Running jfs_debugfs to query the file system usage the same way GParted
does produces an infinite amount of repeating output:

    # echo dm | jfs_debugfs /dev/sdb1

So jfs_debugfs gets stuck in an infinite loop inside the dmap subcommand
when it encounters EOF.  GParted and the read JFS usage test read this
output until memory is exhausted and crash.  This is exactly what was
happening in closed bug 794947.  Even installed jfsutils from Fedora 29
on Fedora 30 and visa versa.  jfs_debugfs still produced an infinite
amount of output on Fedora 30 and worked correctly on Fedora 29.  So
it's not the build of jfsutils, but something in the OS that is making
the difference!

Anyway fix by providing the instruction to exit from the dmap
subcommand, and quit from jfs_debugfs itself, like this:

    # echo -e 'dmap\nx\nquit' | jfs_debugfs /dev/sdb1

Bug 794947 - gparted hangs when sees JFS partition on discovering
             partitions
Closes !49 - Add file system interface tests
2019-11-09 17:18:34 +00:00
Mike Fleetwood c15d0cd6aa Accept FS usage figures within significant unallocated threshold (!49)
So far the read file system usage figures, read via the file system
interface classes using file system specific tools, have been checked to
the exact sector for:
     0 <= used <= size
     0 <= unused <= size
     unallocated = 0
     used + unused = size

However for JFS and NTFS this fails like this:

    # ./test_SupportedFileSystems --gtest_filter='*ReadUsage/*' | fgrep ' ms'
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/btrfs (335 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/exfat (0 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/ext2 (38 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/ext3 (131 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/ext4 (32 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/f2fs (47 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/fat16 (19 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/fat32 (48 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/hfs (0 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/hfsplus (0 ms)
    [  FAILED  ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs, where GetParam() = 17 (73 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/linuxswap (20 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/luks (0 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/lvm2pv (410 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/minix (0 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/nilfs2 (226 ms)
    [  FAILED  ] My/SupportedFileSystemsTest.CreateAndReadUsage/ntfs, where GetParam() = 23 (56 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/reiser4 (49 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/reiserfs (139 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/udf (34 ms)
    [       OK ] My/SupportedFileSystemsTest.CreateAndReadUsage/xfs (67 ms)
    [----------] 21 tests from My/SupportedFileSystemsTest (1726 ms total)
    [==========] 21 tests from 1 test case ran. (1726 ms total)

    # ./test_SupportedFileSystems --gtest_filter='*ReadUsage/jfs:*ReadUsage/ntfs'
    Running main() from test_SupportedFileSystems.cc
    Note: Google Test filter = *ReadUsage/jfs:*ReadUsage/ntfs
    [==========] Running 2 tests from 1 test case.
    [----------] Global test environment set-up.
    [----------] 2 tests from My/SupportedFileSystemsTest
    [ RUN      ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs
    test_SupportedFileSystems.cc:465: Failure
    Expected equality of these values:
      m_partition.sectors_unallocated
        Which is: 2472
      0
    test_SupportedFileSystems.cc:517: Failure
    Expected equality of these values:
      m_partition.sectors_used + m_partition.sectors_unused
        Which is: 521816
      m_partition.get_sector_length()
        Which is: 524288
    [  FAILED  ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs, where GetParam() = 17 (36 ms)
    [ RUN      ] My/SupportedFileSystemsTest.CreateAndReadUsage/ntfs
    test_SupportedFileSystems.cc:465: Failure
    Expected equality of these values:
      m_partition.sectors_unallocated
        Which is: 8
      0
    test_SupportedFileSystems.cc:517: Failure
    Expected equality of these values:
      m_partition.sectors_used + m_partition.sectors_unused
        Which is: 524280
      m_partition.get_sector_length()
        Which is: 524288
    [  FAILED  ] My/SupportedFileSystemsTest.CreateAndReadUsage/ntfs, where GetParam() = 23 (35 ms)
    [----------] 2 tests from My/SupportedFileSystemsTest (71 ms total)

    [----------] Global test environment tear-down
    [==========] 2 tests from 1 test case ran. (72 ms total)
    [  PASSED  ] 0 tests.
    [  FAILED  ] 2 tests, listed below:
    [  FAILED  ] My/SupportedFileSystemsTest.CreateAndReadUsage/jfs, where GetParam() = 17
    [  FAILED  ] My/SupportedFileSystemsTest.CreateAndReadUsage/ntfs, where GetParam() = 23

     2 FAILED TESTS

So JFS is reporting 2472 unallocated sectors in a size of 524288 sectors
and NTFS is reporting 8 unallocated sectors in the same size.  This
exact issue is already solved for GParted so that it doesn't show a
small amount of unallocated space by commits [1][2] from Bug 499202 [3].

Fix the same way, use the accessors to the file system usage figures
which don't show unallocated space when it is below the significant
threshold.

[1] b5c80f18a9
    Enhance calculation of significant unallocated space (#499202)

[2] 7ebedc4bb3
    Don't show intrinsic unallocated space (#499202)

[3] Bug 499202 - gparted does not see the difference if partition size
                 differs from filesystem size
    https://bugzilla.gnome.org/show_bug.cgi?id=499202

Closes !49 - Add file system interface tests
2019-11-09 17:18:34 +00:00