Linux inode Full and error “Can’t create/write to file”

 

From Wiki

“

In a Unix-style file system, an index node, informally referred to as an inode, is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. Each inode stores the attributes and disk block location(s) of the filesystem object's data. Filesystem object attributes may include manipulation metadata (e.g. change, access, modify time), as well as owner and permission data (e.g. group-id,user-id, permissions).

A Linux directory lists other filesystem objects by name, normally identifying the listed object by referring to its inode. The directory contains an entry for itself, its parent, and each of its children.

“

If inode is full, you may face issue with error message similar “Can’t create/write to file”

To check how much inode has already consumed, use the command

# df -i /<directory>

you won’t be able to create file or write to file in that directory if indoe is 100% full.

 

In my case, i saw lots of empty files generated by faulty applications, so i just delete using rm –f command

 

You can also run the following cronjob to keep /<directory> clear.

Run crontab -e and add the following:
0 */4 * * * /usr/sbin/tmpwatch -am 12 /<directory>

Don’t underestimate the RAID controller on your server !!

Dell PERC Controller – RAID card
 
 
 
We have Dell Server PowerEdge R620, running Oracle VM virtualization. We noticed that the disk performance is very poor, much like USB2.0 speed, 22.4 MB/s, this is unacceptable for server.

Our disk configuration is RAID 5, 6x 10K, 6G SAS HDD, 900GB
Upon investigation, we realized the RAID controller card that we purchased, Dell Perc H310 is a budget raid controller. That causes the bottleneck.
Luckily, there is a “drop in replacement” H310 to H710p. Model shown below is H310 mini mono.



 
 
 
 
After i changed the RAID controller to PERC H710p, ran the same test,
 

 
Woww! 139 MB/s – that’s a massive improvements!!
 
 

With the same test, but with 5GB file, 300MB/s


but.... don;t get too excited yet...


You may have a fast 15K or SSD disk but having a good RAID controller is important too. Slow RAID controller with fast disk behind means you are not utilizing the full potential.
Once the card is in, choose you caching method.


There are three main caching techniques that can be deployed, each with their own pros and cons.
· Write-through cache directs write I/O onto cache and through to underlying permanent storage before confirming I/O completion to the host. This ensures data updates are safely stored on, for example, a shared storage array, but has the disadvantage that I/O still experiences latency based on writing to that storage. Write-through cache is good for applications that write and then re-read data frequently as data is stored in cache and results in low read latency.
· Write-around cache is a similar technique to write-through cache, but write I/O is written directly to permanent storage, bypassing the cache. This can reduce the cache being flooded with write I/O that will not subsequently be re-read, but has the disadvantage is that a read request for recently written data will create a “cache miss” and have to be read from slower bulk storage and experience higher latency.
· Write-back cache is where write I/O is directed to cache and completion is immediately confirmed to the host. This results in low latency and high throughput for write-intensive applications, but there is data availability exposure risk because the only copy of the written data is in cache. As we will discuss later, suppliers have added resiliency with products that duplicate writes. Users need to consider whether write-back cache solutions offer enough protection as data is exposed until it is staged to external storage. Write-back cache is the best performing solution for mixed workloads as both read and write I/O have similar response time levels.
 
 

Result after write back cache enabled,

 

double the speed again... it is 572MB/s ... now that's what i called Server speed !!



Reference:
http://en.community.dell.com/support-forums/servers/f/906/t/19533325
http://en.community.dell.com/support-forums/servers/f/906/t/19606391
http://www.computerweekly.com/feature/Write-through-write-around-write-back-Cache-explained

Lync 2013 Control Panel error - 00002098: SecErr: DSID-03150BB9, problem 4003 (INSUFF_ACCESS_RIGHTS)

When trying to edit user in Lync Control Panel 2013, I encounter this error

Active Directory operation failed on "<DC>". You cannot retry this operation: "Insufficient access rights to perform the operation

00002098: SecErr: DSID-03150BB9, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0".You do not have the appropriate permissions to perform this operation in Active Directory. One possible cause is that the Lync Server Control Panel and Remote Windows PowerShell cannot modify users who belong to protected security groups (for example, the Domain Admins group). To manage users in the Domain Admins group, use the Lync Server Management Shell and log on using a Domain Admins account. There are other possible causes. For details, see Lync Server 2010 Help.

Solutions

From dsa.msc (Active Directory Users & Computers) make sure that you are viewing advanced features; from the view menu select Advanced Features.

Find the user in question, edit the users properties, click the security tab, then click advanced. You need to tick the “Include inheritable permissions from this object’s parent” tick box to re propagate the required permissions.

Retry any Lync user changes and this time they should be successful.

Custom report from Dynamics CRM takes long time to load

When triggered custom report from CRM, it takes long time to run. It will display the report but it just takes some time. This is due to CRM and SSRS seeking authentication from Active Directory. If the AD is slow, this will affect the performance.

To solve this issue, make sure the AD is reachable (preferable on the same subnet), then create/set registry key on both CRM and SSRS server for a preffered Domain Controller,

HKLM\Software\Microsoft\MSCRM registry hive

• Name: PreferredDC
• Type: String
• Value: <DC_Name>

Active Directory upgrade 2003 to 2008R2 with error “An Authentication Error Has Occurred. The Encryption Type Requested Is not supported by the KDC”

we experinced that after upgraded the Domain and Forest Level from 2003 to 2008R2, after a day, I cannot connect to my Hyper V Virtual machines with following error ” An Authentication Error Has Occurred. The Encryption Type Requested Is not supported by the KDC”. On the Exchange 2010 server, the transport server stopped and follow error message on application event log,
“ Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=xxxx). Topology discovery failed, error 0×80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message))….”
“Process STORE.EXE (PID=xxxx). All Global Catalog Servers in forest DC=xxx,DC=xx,DC=xx are not responding.”
”Process STORE.EXE (PID=xxxx). All Domain Controller Servers in use are not responding”

A simple resolution to all this trouble is to restart the KERBEROS DISTRIBUTION KEY or KDC service on all Domain controllers. While simply restarting the Service will solve the problem, probably better off just doing a proper restart after upgrading your functional levels, only from 2003 to 2008 / 2008 R2.