Problem
If your StorNext volume is getting a out of disk space error but your file browsers or df
shows you still have plenty of space, then you could be out of Meta space.
Use cvadmin
> show long
and examine the MetaData stripe group, how much Free is available?
Example of 0% free or just 656KB
Stripe Group 0 [MetaFiles] Status:Up,MetaData,Journal,Exclusive Total Blocks:256000 (1000.00 MB) Reserved:0 (0.00 B) Free:164 (656.00 KB) (0%)
We saw in our test that if the Free space is too small (656KB) like in the example above, the expansion process would error out with the following error:
Initializing stripe group MetaFiles_1 information... *Fatal*: Failed to locate free inode *Error*: ERROR *Fatal*: Failed to initialize stripe group Check configuration and try again
In our case we had to delete more files to gain a larger Free space of 5.54MB, this allowed the expansion process to complete without error.
Solution
To solve this problem more Meta space is needed by adding an additional Meta Stripe Group.
See this article for general instruction on the process:
http://support.dvsus.com/hc/en-us/articles/212920426
Note that in the new Meta Stripe Group you must adjust these and other tags accordingly:
index=
name=
diskLabel=
diskType=
ordinal=
and this must be set to false journal="false"
Example of updated .cfgx showing the diskTypes and stripeGroups sections, newly added MetaFiles_1 Stripe Group highlighted:
<diskTypes>
<diskType typeName="MetaDrive" sectors="2048000" sectorSize="512"/>
<diskType typeName="DataDrive" sectors="15627646943" sectorSize="512"/>
</diskTypes>
<stripeGroups>
<stripeGroup index="0" name="MetaFiles" status="up" stripeBreadth="65536" read="true" write="true" metadata="true" journal="true" userdata="false" realTimeIOs="0" realTimeIOsReserve="0" realTimeMB="0" realTimeMBReserve="0" realTimeTokenTimeout="0" multipathMethod="rotate">
<disk index="0" diskLabel="dh_meta03" diskType="MetaDrive" ordinal="0"/>
</stripeGroup>
<stripeGroup index="1" name="DataFiles_0" status="up" stripeBreadth="2097152" read="true" write="true" metadata="false" journal="false" userdata="true" realTimeIOs="0" realTimeIOsReserve="0" realTimeMB="0" realTimeMBReserve="0" realTimeTokenTimeout="0" multipathMethod="rotate">
<disk index="0" diskLabel="lun00" diskType="DataDrive" ordinal="1"/>
<disk index="1" diskLabel="lun01" diskType="DataDrive" ordinal="2"/>
<disk index="2" diskLabel="lun02" diskType="DataDrive" ordinal="3"/>
</stripeGroup>
<stripeGroup index="2" name="MetaFiles_1" status="up" stripeBreadth="65536" read="true" write="true" metadata="true" journal="false" userdata="false" realTimeIOs="0" realTimeIOsReserve="0" realTimeMB="0" realTimeMBReserve="0" realTimeTokenTimeout="0" multipathMethod="rotate">
<disk index="0" diskLabel="dh_meta04" diskType="MetaDrive" ordinal="4"/>
</stripeGroup>
</stripeGroups>
</config>
</configDoc>
Examples
Example of cvupdatefs session:
# cvupdatefs
Checked Build disabled - default.
StorNext File Systems on host mds1-120140002:
1) iron
2) disney
3) DVS-SAN
Choose a file system by number (1-3) or zero (0) to exit -> 1
*Warning*: Inode Expand Min is deprecated and will no longer be supported in future releases
*Warning*: Inode Expand Max is deprecated and will no longer be supported in future releases
*Warning*: Inode Expand Inc is deprecated and will no longer be supported in future releases
The following changes have been detected in the configuration
Please review these changes carefully.
Stripe Group Name Stripe Status MetaData Journal
================= ============= ======== =======
MetaFiles No Change No Change No Change
DataFiles_0 No Change
MetaFiles_1 Create Create
This will modify the file system "iron".
Are you sure you want to continue? [y/N] y
Attempting to acquire arbitration block... successful.
Flushing journal entries... done
Initializing stripe group MetaFiles_1 information... ok
Flushing buffers...
Updating ICB information...
Updating SuperBlock information...
*Warning*: File system 'iron' was modified.
After the expansion process, you will have an additional Meta Stripe Group, see it with cvadmin
> show long
:
Stripe Group 2 [MetaFiles_1] Status:Up,MetaData,Exclusive
Total Blocks:256000 (1000.00 MB) Reserved:0 (0.00 B) Free:255542 (998.21 MB) (99%)
MultiPath Method:Rotate
Stripe Depth:1 Stripe Breadth:16 blocks (64.00 KB)
Realtime limit IO/sec:0 (~0 mb/sec) Non-Realtime reserve IO/sec:0
Committed RTIO/sec:0 Non-RTIO clients:0 Non-RTIO hint IO/sec:0
Disk stripes:
Primary Stripe [MetaFiles_1] Read:Enabled Write:Enabled
Node 0 [dh_meta04]
Note
For this example, we used very small Meta disks of just 1GB (1000.0 MB) to easier illustrate the problem and the solution. Normally the Meta disk should be in the 100+GB, 300GB is better.
0 Comments