A question about SD-Card write on MSP-EXP430F5529

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

A question about SD-Card write on MSP-EXP430F5529

ravim
Hello Everyone,

We are working on a simple data logging application using MSP-EXP430F5529 and Ez430-RF2500 devices using mspgcc. Thanks to excellent resources and this forum, everything is working fine. I had a simple question about SD-Card write.

Our application writes a data packet  to SD-Card as soon as it receive on EXP board. The packet size is 40Bytes and may increase more. The binary data is stored in files on SD card which we later run through our interpreter. Now if someone pulls the SD-Card in midst of write operation what will happen?

I wanted to ask -  is it possible that portion of data packet may be written to file when SD card is suddenly pulled out? Because if this happens the whole series of binary data will not make any sense or may give us totally wrong information.

If this happens how do we handle it? Is there a way to make this write operation atomic??
Reply | Threaded
Open this post in threaded view
|

Re: A question about SD-Card write on MSP-EXP430F5529

Daniel Beer-3
On Wed, Jun 05, 2013 at 07:33:31PM -0700, ravim wrote:

> Our application writes a data packet  to SD-Card as soon as it receive on
> EXP board. The packet size is 40Bytes and may increase more. The binary data
> is stored in files on SD card which we later run through our interpreter.
> Now if someone pulls the SD-Card in midst of write operation what will
> happen?
>
> I wanted to ask -  is it possible that portion of data packet may be written
> to file when SD card is suddenly pulled out? Because if this happens the
> whole series of binary data will not make any sense or may give us totally
> wrong information.
>
> If this happens how do we handle it? Is there a way to make this write
> operation atomic??

If you mean that you're writing a sequence sectors to an SD-card, you
could ensure that your sectors contain a checksum. If the write fails,
the checksum will (with high probability) not match, and you can ignore
the sector.

This assumes that a write failure affects only the sector to which
you're writing. A sensible flash translation layer would hopefully
provide this guarantee.

If you're writing to a file on a VFAT filesystem, it's different. You
might be able to implement a transaction system by making use of the
redundant FAT copies and doing shadow-paging with unused clusters. This
would only guarantee consistency when reading from your own application
though.

Cheers,
Daniel

--
Daniel Beer <[hidden email]>    www.dlbeer.co.nz
IRC: inittab (Freenode)    PGP key: 2048D/160A553B

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Mspgcc-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mspgcc-users
Reply | Threaded
Open this post in threaded view
|

Re: A question about SD-Card write on MSP-EXP430F5529

Sergei Sharonov-2
In reply to this post by ravim
On 6/5/2013 9:33 PM, ravim wrote:

> Hello Everyone,
>
> We are working on a simple data logging application using MSP-EXP430F5529
> and Ez430-RF2500 devices using mspgcc. Thanks to excellent resources and
> this forum, everything is working fine. I had a simple question about
> SD-Card write.
>
> Our application writes a data packet  to SD-Card as soon as it receive on
> EXP board. The packet size is 40Bytes and may increase more. The binary data
> is stored in files on SD card which we later run through our interpreter.
> Now if someone pulls the SD-Card in midst of write operation what will
> happen?
>
> I wanted to ask -  is it possible that portion of data packet may be written
> to file when SD card is suddenly pulled out? Because if this happens the
> whole series of binary data will not make any sense or may give us totally
> wrong information.
>
> If this happens how do we handle it? Is there a way to make this write
> operation atomic??
You are about to open a can of worms. You will have to worry about three
things:
1. Flash translation layer integrity.
Nothing you can do here except finding a supplier that guarantees it. I
heard Swissbit
makes some such guarantees. With a random supplier you are likely to end
up with
a dead SD card if you keep yanking power.
2. Filesystem integrity.
For example FAT is not power fail safe. It does not do changes to file
and file allocation
table in atomic fashion. There are journaling filesystems such as ext3
that are
designed to handle power failures but the size and complexity is beyond
what MSP430
can handle.
3. Atomicity of write operation.
With a few exception filesystems make no guarantees as to atomicity of
write operation.
This is up to user code to implement it. For example, rename operation
is usually
atomic and you can take advantage of that.

Regards,

Sergei

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Mspgcc-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mspgcc-users
Reply | Threaded
Open this post in threaded view
|

Re: A question about SD-Card write on MSP-EXP430F5529

David W. Schultz
In reply to this post by ravim
On 06/05/2013 09:33 PM, ravim wrote:
> I wanted to ask -  is it possible that portion of data packet may be written
> to file when SD card is suddenly pulled out? Because if this happens the
> whole series of binary data will not make any sense or may give us totally
> wrong information.
>
> If this happens how do we handle it? Is there a way to make this write
> operation atomic??
>

All writes are of 512 byte blocks. The SD card will buffer the data you
transfer to it and only once you have sent a full block will it begin
its internal write operation.

So if the card is pulled in the middle of the transfer, then all data in
that block is lost. If it is pulled after the transfer is complete but
before the internal write operation is finished, then the results are
unknown.

If users pulling out a card in the middle of an operation is expected to
be a problem then you must use a data structure that will tolerate that.

--
David W. Schultz
http://home.earthlink.net/~david.schultz
Returned for Regrooving



------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Mspgcc-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mspgcc-users
Reply | Threaded
Open this post in threaded view
|

Re: A question about SD-Card write on MSP-EXP430F5529

Sergei Sharonov-2
On 6/6/2013 6:40 PM, David W. Schultz wrote:
> If users pulling out a card in the middle of an operation is expected
> to be a problem then you must use a data structure that will tolerate
> that.
If a power is pulled from SD card while you are writing to it, in
general you may expect to end up with a corrupted card. You
have no idea what flash translation layer is doing behind the closed
doors. On top of that, a non-journaling filesystem such as FAT
may get corrupted as well.
The FTL behavior will vary from vendor to vendor. Some are
are better then others in maintaining data integrity.

Regards,

Sergei

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Mspgcc-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mspgcc-users
Reply | Threaded
Open this post in threaded view
|

Re: A question about SD-Card write on MSP-EXP430F5529

ravim
Thank you gentlemen for the responses.

I figured out following things I will do since I have a deadline..

1. Choose more reliable SD card (I am still looking for options)
2. Introducing button interrupt to stop writing process and then blinking LEDs for to make aware of safe ejection. I will make sure that every stakeholder follow the process.
3. Data integrity - using checksum on sectors and CRC on data packets ( CRC I have already done, checksum I am in process of implementing)
4. Multiple copies of data packets will be sent to make system more reliable.

I really appreciate your responses and time.
Thanks and regards
Ravi
Reply | Threaded
Open this post in threaded view
|

Re: A question about SD-Card write on MSP-EXP430F5529

Peter Bigot-4
Curious about your motivation for (4) multiple copies.  If you don't
succeed with (3) data integrity, you won't be able to tell that you need a
different copy.  If you do, you'll know on the first read whether that copy
is valid: if it is you don't need the others, and if it isn't then the
others probably got corrupted by the removal of the SD card during or
before the first write so you can't use them either.  Seems (4) doesn't add
value.

Peter


On Fri, Jun 7, 2013 at 11:32 AM, ravim <[hidden email]> wrote:

> Thank you gentlemen for the responses.
>
> I figured out following things I will do since I have a deadline..
>
> 1. Choose more reliable SD card (I am still looking for options)
> 2. Introducing button interrupt to stop writing process and then blinking
> LEDs for to make aware of safe ejection. I will make sure that every
> stakeholder follow the process.
> 3. Data integrity - using checksum on sectors and CRC on data packets ( CRC
> I have already done, checksum I am in process of implementing)
> 4. Multiple copies of data packets will be sent to make system more
> reliable.
>
> I really appreciate your responses and time.
> Thanks and regards
> Ravi
>
>
>
> --
> View this message in context:
> http://msp430-gcc-users.1086195.n5.nabble.com/A-question-about-SD-Card-write-on-MSP-EXP430F5529-tp6710p6715.html
> Sent from the MSP430 gcc - Users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. A cloud service to automate IT design, transition and operations
> 2. Dashboards that offer high-level views of enterprise services
> 3. A single system of record for all IT processes
> http://p.sf.net/sfu/servicenow-d2d-j
> _______________________________________________
> Mspgcc-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/mspgcc-users
>

------------------------------------------------------------------------------
How ServiceNow helps IT people transform IT departments:
1. A cloud service to automate IT design, transition and operations
2. Dashboards that offer high-level views of enterprise services
3. A single system of record for all IT processes
http://p.sf.net/sfu/servicenow-d2d-j
_______________________________________________
Mspgcc-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/mspgcc-users